2024-09-15T20:10:02.734 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T20:10:02.745 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T20:10:02.832 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905555 branch: main description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} email: ceph-qa@ceph.com first_in_suite: false job_id: '7905555' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: main ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 workunit: branch: main sha1: 7498493279982dfe87f00616198a5967475f6169 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2855 sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 sleep_before_teardown: 0 subset: 17936/100000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 7498493279982dfe87f00616198a5967475f6169 targets: smithi139.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIyBIxMbutuunx0JNlNRREtWqmay3Kr5nMVifEJljcjQ1bf21Yrxc7zod+vnMdHtPby7clIN8uTDQAoSiXxk+Tg= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_20:00:16 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T20:10:02.832 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa; will attempt to use it 2024-09-15T20:10:02.833 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks 2024-09-15T20:10:02.833 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T20:10:02.834 INFO:teuthology.task.internal:Checking packages... 2024-09-15T20:10:02.853 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'fc3e0bff7468fd55478adbae7f75508b749124b6' 2024-09-15T20:10:02.853 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T20:10:02.854 INFO:teuthology.packaging:ref: None 2024-09-15T20:10:02.854 INFO:teuthology.packaging:tag: None 2024-09-15T20:10:02.854 INFO:teuthology.packaging:branch: main 2024-09-15T20:10:02.854 INFO:teuthology.packaging:sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T20:10:02.854 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=main 2024-09-15T20:10:03.051 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4930-gfc3e0bff-1jammy 2024-09-15T20:10:03.052 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T20:10:03.061 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T20:10:03.061 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T20:10:03.078 INFO:teuthology.task.internal:Saving configuration 2024-09-15T20:10:03.086 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T20:10:03.097 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T20:10:03.121 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi139.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905555', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 20:03:28.230853', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKbuHDeaolEAYcVxdMx7R/1JLQYBZwJGnqpAEvOa692Q'} 2024-09-15T20:10:03.121 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T20:10:03.133 INFO:teuthology.task.internal:roles: ubuntu@smithi139.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-15T20:10:03.133 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T20:10:03.171 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd178c06ef0>, signals=[15]) 2024-09-15T20:10:03.171 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T20:10:03.179 INFO:teuthology.task.internal:Opening connections... 2024-09-15T20:10:03.179 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi139.front.sepia.ceph.com 2024-09-15T20:10:03.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:10:03.259 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T20:10:03.268 DEBUG:teuthology.orchestra.run.smithi139:> uname -m 2024-09-15T20:10:03.361 INFO:teuthology.orchestra.run.smithi139.stdout:x86_64 2024-09-15T20:10:03.361 DEBUG:teuthology.orchestra.run.smithi139:> cat /etc/os-release 2024-09-15T20:10:03.409 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-15T20:10:03.409 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="Ubuntu" 2024-09-15T20:10:03.409 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="22.04" 2024-09-15T20:10:03.409 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-15T20:10:03.409 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_CODENAME=jammy 2024-09-15T20:10:03.409 INFO:teuthology.orchestra.run.smithi139.stdout:ID=ubuntu 2024-09-15T20:10:03.409 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE=debian 2024-09-15T20:10:03.410 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-15T20:10:03.410 INFO:teuthology.orchestra.run.smithi139.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-15T20:10:03.410 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-15T20:10:03.410 INFO:teuthology.orchestra.run.smithi139.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-15T20:10:03.410 INFO:teuthology.orchestra.run.smithi139.stdout:UBUNTU_CODENAME=jammy 2024-09-15T20:10:03.410 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server 2024-09-15T20:10:03.427 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T20:10:03.436 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T20:10:03.448 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T20:10:03.448 DEBUG:teuthology.orchestra.run.smithi139:> test '!' -e /home/ubuntu/cephtest 2024-09-15T20:10:03.457 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T20:10:03.463 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T20:10:03.463 DEBUG:teuthology.orchestra.run.smithi139:> test -z $(ls -A /var/lib/ceph) 2024-09-15T20:10:03.507 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T20:10:03.544 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T20:10:03.562 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-15T20:10:03.562 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-15T20:10:03.563 DEBUG:teuthology.orchestra.run.smithi139:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T20:10:03.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:10:03.567 DEBUG:teuthology.orchestra.run.smithi139:> uname -r 2024-09-15T20:10:03.613 INFO:teuthology.orchestra.run.smithi139.stdout:5.15.0-56-generic 2024-09-15T20:10:03.614 INFO:teuthology.task.kernel:Running kernel on smithi139: 5.15.0-56-generic 2024-09-15T20:10:03.614 DEBUG:teuthology.orchestra.run.smithi139:> sudo apt-get clean 2024-09-15T20:10:03.765 DEBUG:teuthology.orchestra.run.smithi139:> sudo apt-get update 2024-09-15T20:10:03.921 INFO:teuthology.orchestra.run.smithi139.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-15T20:10:04.034 INFO:teuthology.orchestra.run.smithi139.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-15T20:10:04.115 INFO:teuthology.orchestra.run.smithi139.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-15T20:10:04.128 INFO:teuthology.orchestra.run.smithi139.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-15T20:10:04.229 INFO:teuthology.orchestra.run.smithi139.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-15T20:10:04.233 INFO:teuthology.orchestra.run.smithi139.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-15T20:10:04.235 INFO:teuthology.orchestra.run.smithi139.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-15T20:10:04.235 INFO:teuthology.orchestra.run.smithi139.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-15T20:10:04.235 INFO:teuthology.orchestra.run.smithi139.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-15T20:10:04.255 INFO:teuthology.orchestra.run.smithi139.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-15T20:10:04.258 INFO:teuthology.orchestra.run.smithi139.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-15T20:10:04.258 INFO:teuthology.orchestra.run.smithi139.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-15T20:10:04.262 INFO:teuthology.orchestra.run.smithi139.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-15T20:10:04.267 INFO:teuthology.orchestra.run.smithi139.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-15T20:10:04.269 INFO:teuthology.orchestra.run.smithi139.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-15T20:10:04.269 INFO:teuthology.orchestra.run.smithi139.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-15T20:10:04.270 INFO:teuthology.orchestra.run.smithi139.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-15T20:10:04.270 INFO:teuthology.orchestra.run.smithi139.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-15T20:10:04.321 INFO:teuthology.orchestra.run.smithi139.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-15T20:10:04.469 INFO:teuthology.orchestra.run.smithi139.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-15T20:10:04.606 INFO:teuthology.orchestra.run.smithi139.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-15T20:10:04.801 INFO:teuthology.orchestra.run.smithi139.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-15T20:10:04.962 INFO:teuthology.orchestra.run.smithi139.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-15T20:10:04.977 INFO:teuthology.orchestra.run.smithi139.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-15T20:10:04.977 INFO:teuthology.orchestra.run.smithi139.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-15T20:10:05.046 INFO:teuthology.orchestra.run.smithi139.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-15T20:10:05.047 INFO:teuthology.orchestra.run.smithi139.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-15T20:10:05.064 INFO:teuthology.orchestra.run.smithi139.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-15T20:10:05.065 INFO:teuthology.orchestra.run.smithi139.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-15T20:10:05.095 INFO:teuthology.orchestra.run.smithi139.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-15T20:10:05.113 INFO:teuthology.orchestra.run.smithi139.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-15T20:10:05.118 INFO:teuthology.orchestra.run.smithi139.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-15T20:10:05.125 INFO:teuthology.orchestra.run.smithi139.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-15T20:10:05.126 INFO:teuthology.orchestra.run.smithi139.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-15T20:10:05.197 INFO:teuthology.orchestra.run.smithi139.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-15T20:10:05.197 INFO:teuthology.orchestra.run.smithi139.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-15T20:10:05.197 INFO:teuthology.orchestra.run.smithi139.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-15T20:10:05.199 INFO:teuthology.orchestra.run.smithi139.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-15T20:10:05.200 INFO:teuthology.orchestra.run.smithi139.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-15T20:10:05.207 INFO:teuthology.orchestra.run.smithi139.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-15T20:10:05.207 INFO:teuthology.orchestra.run.smithi139.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-15T20:10:05.207 INFO:teuthology.orchestra.run.smithi139.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-15T20:10:05.288 INFO:teuthology.orchestra.run.smithi139.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-15T20:10:05.288 INFO:teuthology.orchestra.run.smithi139.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-15T20:10:10.773 INFO:teuthology.orchestra.run.smithi139.stdout:Fetched 16.0 MB in 2s (7,365 kB/s) 2024-09-15T20:10:12.246 INFO:teuthology.orchestra.run.smithi139.stdout:Reading package lists... 2024-09-15T20:10:12.264 DEBUG:teuthology.orchestra.run.smithi139:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T20:10:12.328 INFO:teuthology.orchestra.run.smithi139.stdout:Reading package lists... 2024-09-15T20:10:12.539 INFO:teuthology.orchestra.run.smithi139.stdout:Building dependency tree... 2024-09-15T20:10:12.540 INFO:teuthology.orchestra.run.smithi139.stdout:Reading state information... 2024-09-15T20:10:12.715 INFO:teuthology.orchestra.run.smithi139.stdout:The following packages were automatically installed and are no longer required: 2024-09-15T20:10:12.715 INFO:teuthology.orchestra.run.smithi139.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-15T20:10:12.715 INFO:teuthology.orchestra.run.smithi139.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-15T20:10:12.715 INFO:teuthology.orchestra.run.smithi139.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-15T20:10:12.716 INFO:teuthology.orchestra.run.smithi139.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-15T20:10:12.716 INFO:teuthology.orchestra.run.smithi139.stdout: python2.7-minimal 2024-09-15T20:10:12.716 INFO:teuthology.orchestra.run.smithi139.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-15T20:10:12.717 INFO:teuthology.orchestra.run.smithi139.stdout:The following additional packages will be installed: 2024-09-15T20:10:12.717 INFO:teuthology.orchestra.run.smithi139.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-15T20:10:12.717 INFO:teuthology.orchestra.run.smithi139.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-15T20:10:12.717 INFO:teuthology.orchestra.run.smithi139.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-15T20:10:12.719 INFO:teuthology.orchestra.run.smithi139.stdout:Suggested packages: 2024-09-15T20:10:12.719 INFO:teuthology.orchestra.run.smithi139.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-15T20:10:12.719 INFO:teuthology.orchestra.run.smithi139.stdout:Recommended packages: 2024-09-15T20:10:12.719 INFO:teuthology.orchestra.run.smithi139.stdout: thermald 2024-09-15T20:10:12.761 INFO:teuthology.orchestra.run.smithi139.stdout:The following NEW packages will be installed: 2024-09-15T20:10:12.761 INFO:teuthology.orchestra.run.smithi139.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-15T20:10:12.761 INFO:teuthology.orchestra.run.smithi139.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-15T20:10:12.762 INFO:teuthology.orchestra.run.smithi139.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-15T20:10:12.763 INFO:teuthology.orchestra.run.smithi139.stdout:The following packages will be upgraded: 2024-09-15T20:10:12.763 INFO:teuthology.orchestra.run.smithi139.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-15T20:10:12.814 INFO:teuthology.orchestra.run.smithi139.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-15T20:10:12.814 INFO:teuthology.orchestra.run.smithi139.stdout:Need to get 113 MB of archives. 2024-09-15T20:10:12.814 INFO:teuthology.orchestra.run.smithi139.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-15T20:10:12.814 INFO:teuthology.orchestra.run.smithi139.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-15T20:10:13.275 INFO:teuthology.orchestra.run.smithi139.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-15T20:10:13.389 INFO:teuthology.orchestra.run.smithi139.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-15T20:10:14.221 INFO:teuthology.orchestra.run.smithi139.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-15T20:10:14.221 INFO:teuthology.orchestra.run.smithi139.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-15T20:10:14.221 INFO:teuthology.orchestra.run.smithi139.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-15T20:10:14.343 INFO:teuthology.orchestra.run.smithi139.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-15T20:10:14.365 INFO:teuthology.orchestra.run.smithi139.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-15T20:10:14.757 INFO:teuthology.orchestra.run.smithi139.stdout:Fetched 113 MB in 2s (71.1 MB/s) 2024-09-15T20:10:15.094 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-15T20:10:17.064 INFO:teuthology.orchestra.run.smithi139.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-15T20:10:17.069 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-15T20:10:17.149 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:10:19.342 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-15T20:10:19.363 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-15T20:10:19.450 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:10:19.945 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-15T20:10:19.965 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-15T20:10:20.003 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:10:26.770 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-15T20:10:26.895 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-15T20:10:27.228 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-15T20:10:27.366 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-15T20:10:27.660 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-15T20:10:27.683 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-15T20:10:27.718 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-15T20:10:33.231 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-15T20:10:33.268 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-15T20:10:33.306 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:10:35.961 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-15T20:10:36.083 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-15T20:10:36.449 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-15T20:10:36.554 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:10:36.696 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-15T20:10:36.824 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:10:38.162 INFO:teuthology.orchestra.run.smithi139.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-15T20:10:38.162 INFO:teuthology.orchestra.run.smithi139.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-15T20:10:38.162 INFO:teuthology.orchestra.run.smithi139.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-15T20:10:38.162 INFO:teuthology.orchestra.run.smithi139.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-15T20:10:38.327 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:10:40.278 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:10:41.696 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-15T20:10:41.846 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-15T20:10:41.973 INFO:teuthology.orchestra.run.smithi139.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:10:42.041 INFO:teuthology.orchestra.run.smithi139.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-15T20:10:42.041 INFO:teuthology.orchestra.run.smithi139.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-15T20:10:56.870 INFO:teuthology.orchestra.run.smithi139.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-15T20:10:56.870 INFO:teuthology.orchestra.run.smithi139.stdout:Sourcing file `/etc/default/grub' 2024-09-15T20:10:56.898 INFO:teuthology.orchestra.run.smithi139.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T20:10:56.913 INFO:teuthology.orchestra.run.smithi139.stdout:Generating grub configuration file ... 2024-09-15T20:10:57.214 INFO:teuthology.orchestra.run.smithi139.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-15T20:10:57.230 INFO:teuthology.orchestra.run.smithi139.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-15T20:10:57.573 INFO:teuthology.orchestra.run.smithi139.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-15T20:10:57.578 INFO:teuthology.orchestra.run.smithi139.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-15T20:10:57.642 INFO:teuthology.orchestra.run.smithi139.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-15T20:10:57.647 INFO:teuthology.orchestra.run.smithi139.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-15T20:10:57.829 INFO:teuthology.orchestra.run.smithi139.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-15T20:10:57.829 INFO:teuthology.orchestra.run.smithi139.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-15T20:10:57.829 INFO:teuthology.orchestra.run.smithi139.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-15T20:10:57.865 INFO:teuthology.orchestra.run.smithi139.stdout:done 2024-09-15T20:10:58.350 DEBUG:teuthology.orchestra.run.smithi139:> dpkg -s linux-image-generic 2024-09-15T20:10:58.380 INFO:teuthology.orchestra.run.smithi139.stdout:Package: linux-image-generic 2024-09-15T20:10:58.380 INFO:teuthology.orchestra.run.smithi139.stdout:Status: install ok installed 2024-09-15T20:10:58.380 INFO:teuthology.orchestra.run.smithi139.stdout:Priority: optional 2024-09-15T20:10:58.381 INFO:teuthology.orchestra.run.smithi139.stdout:Section: kernel 2024-09-15T20:10:58.381 INFO:teuthology.orchestra.run.smithi139.stdout:Installed-Size: 21 2024-09-15T20:10:58.381 INFO:teuthology.orchestra.run.smithi139.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T20:10:58.381 INFO:teuthology.orchestra.run.smithi139.stdout:Architecture: amd64 2024-09-15T20:10:58.381 INFO:teuthology.orchestra.run.smithi139.stdout:Source: linux-meta 2024-09-15T20:10:58.381 INFO:teuthology.orchestra.run.smithi139.stdout:Version: 5.15.0.121.121 2024-09-15T20:10:58.381 INFO:teuthology.orchestra.run.smithi139.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-15T20:10:58.381 INFO:teuthology.orchestra.run.smithi139.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T20:10:58.381 INFO:teuthology.orchestra.run.smithi139.stdout:Recommends: thermald 2024-09-15T20:10:58.381 INFO:teuthology.orchestra.run.smithi139.stdout:Description: Generic Linux kernel image 2024-09-15T20:10:58.381 INFO:teuthology.orchestra.run.smithi139.stdout: This package will always depend on the latest generic kernel image 2024-09-15T20:10:58.381 INFO:teuthology.orchestra.run.smithi139.stdout: available. 2024-09-15T20:10:58.381 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-15T20:10:58.382 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T20:10:58.382 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T20:10:58.382 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-15T20:10:58.382 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi139.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T20:10:58.382 DEBUG:teuthology.orchestra.run.smithi139:> sudo apt-get clean 2024-09-15T20:10:58.570 DEBUG:teuthology.orchestra.run.smithi139:> sudo apt-get update 2024-09-15T20:10:58.722 INFO:teuthology.orchestra.run.smithi139.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-15T20:10:58.722 INFO:teuthology.orchestra.run.smithi139.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-15T20:10:58.736 INFO:teuthology.orchestra.run.smithi139.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-15T20:10:58.830 INFO:teuthology.orchestra.run.smithi139.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-15T20:11:00.160 INFO:teuthology.orchestra.run.smithi139.stdout:Reading package lists... 2024-09-15T20:11:00.179 DEBUG:teuthology.orchestra.run.smithi139:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T20:11:00.241 INFO:teuthology.orchestra.run.smithi139.stdout:Reading package lists... 2024-09-15T20:11:00.465 INFO:teuthology.orchestra.run.smithi139.stdout:Building dependency tree... 2024-09-15T20:11:00.465 INFO:teuthology.orchestra.run.smithi139.stdout:Reading state information... 2024-09-15T20:11:00.645 INFO:teuthology.orchestra.run.smithi139.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-15T20:11:00.645 INFO:teuthology.orchestra.run.smithi139.stdout:The following packages were automatically installed and are no longer required: 2024-09-15T20:11:00.645 INFO:teuthology.orchestra.run.smithi139.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-15T20:11:00.645 INFO:teuthology.orchestra.run.smithi139.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-15T20:11:00.645 INFO:teuthology.orchestra.run.smithi139.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-15T20:11:00.646 INFO:teuthology.orchestra.run.smithi139.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-15T20:11:00.646 INFO:teuthology.orchestra.run.smithi139.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-15T20:11:00.646 INFO:teuthology.orchestra.run.smithi139.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-15T20:11:00.685 INFO:teuthology.orchestra.run.smithi139.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-15T20:11:00.687 DEBUG:teuthology.orchestra.run.smithi139:> dpkg -s linux-image-generic 2024-09-15T20:11:00.700 INFO:teuthology.orchestra.run.smithi139.stdout:Package: linux-image-generic 2024-09-15T20:11:00.700 INFO:teuthology.orchestra.run.smithi139.stdout:Status: install ok installed 2024-09-15T20:11:00.700 INFO:teuthology.orchestra.run.smithi139.stdout:Priority: optional 2024-09-15T20:11:00.700 INFO:teuthology.orchestra.run.smithi139.stdout:Section: kernel 2024-09-15T20:11:00.700 INFO:teuthology.orchestra.run.smithi139.stdout:Installed-Size: 21 2024-09-15T20:11:00.700 INFO:teuthology.orchestra.run.smithi139.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T20:11:00.700 INFO:teuthology.orchestra.run.smithi139.stdout:Architecture: amd64 2024-09-15T20:11:00.700 INFO:teuthology.orchestra.run.smithi139.stdout:Source: linux-meta 2024-09-15T20:11:00.700 INFO:teuthology.orchestra.run.smithi139.stdout:Version: 5.15.0.121.121 2024-09-15T20:11:00.700 INFO:teuthology.orchestra.run.smithi139.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-15T20:11:00.700 INFO:teuthology.orchestra.run.smithi139.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T20:11:00.700 INFO:teuthology.orchestra.run.smithi139.stdout:Recommends: thermald 2024-09-15T20:11:00.701 INFO:teuthology.orchestra.run.smithi139.stdout:Description: Generic Linux kernel image 2024-09-15T20:11:00.701 INFO:teuthology.orchestra.run.smithi139.stdout: This package will always depend on the latest generic kernel image 2024-09-15T20:11:00.701 INFO:teuthology.orchestra.run.smithi139.stdout: available. 2024-09-15T20:11:00.701 DEBUG:teuthology.orchestra.run.smithi139:> mktemp 2024-09-15T20:11:00.747 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.wfIPGrzmDC 2024-09-15T20:11:00.748 DEBUG:teuthology.orchestra.run.smithi139:> sudo cp /boot/grub/grub.cfg /tmp/tmp.wfIPGrzmDC 2024-09-15T20:11:00.796 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.wfIPGrzmDC 2024-09-15T20:11:00.911 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.wfIPGrzmDC is 10KB 2024-09-15T20:11:00.924 DEBUG:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.wfIPGrzmDC 2024-09-15T20:11:00.928 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-15T20:11:00.982 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-15T20:11:00.983 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-15T20:11:00.983 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-15T20:11:01.046 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-15T20:11:01.047 DEBUG:teuthology.orchestra.run.smithi139:> sudo update-grub 2024-09-15T20:11:02.282 INFO:teuthology.orchestra.run.smithi139.stderr:Sourcing file `/etc/default/grub' 2024-09-15T20:11:02.283 INFO:teuthology.orchestra.run.smithi139.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T20:11:02.287 INFO:teuthology.orchestra.run.smithi139.stderr:Generating grub configuration file ... 2024-09-15T20:11:02.485 INFO:teuthology.orchestra.run.smithi139.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-15T20:11:02.491 INFO:teuthology.orchestra.run.smithi139.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-15T20:11:02.724 INFO:teuthology.orchestra.run.smithi139.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-15T20:11:02.729 INFO:teuthology.orchestra.run.smithi139.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-15T20:11:02.770 INFO:teuthology.orchestra.run.smithi139.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-15T20:11:02.776 INFO:teuthology.orchestra.run.smithi139.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-15T20:11:02.908 INFO:teuthology.orchestra.run.smithi139.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-15T20:11:02.908 INFO:teuthology.orchestra.run.smithi139.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-15T20:11:02.908 INFO:teuthology.orchestra.run.smithi139.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-15T20:11:02.918 INFO:teuthology.orchestra.run.smithi139.stderr:done 2024-09-15T20:11:02.920 DEBUG:teuthology.orchestra.run.smithi139:> sudo shutdown -r now 2024-09-15T20:11:32.947 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-15T20:11:32.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-15T20:11:32.948 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:11:51.485 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.139 2024-09-15T20:12:00.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-15T20:12:00.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:12:03.549 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.139 2024-09-15T20:12:15.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-15T20:12:15.559 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:12:33.981 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.139 2024-09-15T20:12:48.994 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-15T20:12:48.994 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:12:49.219 DEBUG:teuthology.orchestra.run.smithi139:> true 2024-09-15T20:12:49.834 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-15T20:12:49.834 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-15T20:12:49.834 DEBUG:teuthology.orchestra.run.smithi139:> uname -r 2024-09-15T20:12:49.882 INFO:teuthology.orchestra.run.smithi139.stdout:5.15.0-121-generic 2024-09-15T20:12:49.882 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-15T20:12:49.882 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T20:12:49.882 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-15T20:12:50.883 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-15T20:12:50.884 DEBUG:teuthology.orchestra.run.smithi139:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T20:12:51.232 INFO:teuthology.orchestra.run.smithi139.stdout:ttyS1 2024-09-15T20:12:51.268 DEBUG:teuthology.parallel:result is None 2024-09-15T20:12:51.268 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T20:12:51.274 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T20:12:51.274 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T20:12:51.280 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T20:12:51.287 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T20:12:51.297 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T20:12:51.297 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T20:12:51.341 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T20:12:51.347 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T20:12:51.347 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T20:12:51.402 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:12:51.410 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:12:51.412 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T20:12:51.418 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T20:12:51.418 DEBUG:teuthology.orchestra.run.smithi139:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T20:12:51.466 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T20:12:51.475 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T20:12:51.475 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T20:12:51.514 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T20:12:51.562 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T20:12:51.610 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-15T20:12:51.610 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T20:12:51.679 DEBUG:teuthology.orchestra.run.smithi139:> sudo service rsyslog restart 2024-09-15T20:12:51.753 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T20:12:51.760 INFO:teuthology.task.internal:Starting timer... 2024-09-15T20:12:51.760 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T20:12:51.843 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T20:12:51.851 DEBUG:teuthology.task.selinux:Excluding smithi139: OS 'ubuntu' does not support SELinux 2024-09-15T20:12:51.851 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T20:12:51.851 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-15T20:12:51.851 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T20:12:51.851 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T20:12:51.866 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-15T20:12:51.869 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-15T20:12:51.976 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T20:12:51.986 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T20:12:51.987 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi139.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T20:20:23.831 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi139.front.sepia.ceph.com')] 2024-09-15T20:20:23.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-15T20:20:23.834 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:20:24.044 DEBUG:teuthology.orchestra.run.smithi139:> true 2024-09-15T20:20:24.111 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com' 2024-09-15T20:20:24.111 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T20:20:24.132 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T20:20:24.132 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T20:20:24.133 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T20:20:24.181 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-15T20:20:24.181 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: Command line: ntpd -gq 2024-09-15T20:20:24.181 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: ---------------------------------------------------- 2024-09-15T20:20:24.182 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: ntp-4 is maintained by Network Time Foundation, 2024-09-15T20:20:24.182 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-15T20:20:24.182 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: corporation. Support and training for ntp-4 are 2024-09-15T20:20:24.182 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: available at https://www.nwtime.org/support 2024-09-15T20:20:24.182 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: ---------------------------------------------------- 2024-09-15T20:20:24.182 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: proto: precision = 0.058 usec (-24) 2024-09-15T20:20:24.182 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: basedate set to 2022-02-04 2024-09-15T20:20:24.183 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: gps base set to 2022-02-06 (week 2196) 2024-09-15T20:20:24.183 INFO:teuthology.orchestra.run.smithi139.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T20:20:24.183 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T20:20:24.183 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: restrict ::: KOD does nothing without LIMITED. 2024-09-15T20:20:24.184 INFO:teuthology.orchestra.run.smithi139.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-15T20:20:24.184 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: Listen and drop on 0 v6wildcard [::]:123 2024-09-15T20:20:24.185 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-15T20:20:24.185 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: Listen normally on 2 lo 127.0.0.1:123 2024-09-15T20:20:24.185 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: Listen normally on 3 enp3s0f1 172.21.15.139:123 2024-09-15T20:20:24.185 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: Listen normally on 4 lo [::1]:123 2024-09-15T20:20:24.185 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f59%5]:123 2024-09-15T20:20:24.185 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:24 ntpd[18492]: Listening on routing socket on fd #22 for interface updates 2024-09-15T20:20:25.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:25 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:25.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:25 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:25.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:25 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:25.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:25 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:25.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:25 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:25.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:25 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:25.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:25 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:20:25.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:25 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:20:25.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:25 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:25.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:25 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:25.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:25 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:20:25.183 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:25 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:20:26.181 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:26 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:26.181 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:26 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:26.181 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:26 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:26.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:26 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:26.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:26 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:20:26.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:26 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:20:27.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:27 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:27.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:27 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:27.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:27 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:27.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:27 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:27.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:27 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:27.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:27 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:27.183 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:27 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:27.183 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:27 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:27.183 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:27 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:20:27.183 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:27 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:20:27.183 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:27 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:27.183 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:27 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:27.183 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:27 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:20:27.183 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:27 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:20:28.181 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:28 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:28.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:28 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:28.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:28 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:28.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:28 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:28.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:28 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:20:28.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:28 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:20:29.181 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:29 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:29 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:29 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:29 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:29 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:29 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:29 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:20:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:29 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:20:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:29 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:29 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:29 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:20:29.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:29 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:20:30.183 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:30 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:30.183 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:30 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:30.183 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:30 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:30.183 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:30 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:30.183 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:30 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:20:30.183 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:30 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:20:31.181 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:31 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:31.181 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:31 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:31.181 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:31 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:31.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:31 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:31.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:31 ntpd[18492]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:20:31.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:31 ntpd[18492]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:20:31.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:31 ntpd[18492]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:20:31.182 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:31 ntpd[18492]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:20:31.182 INFO:teuthology.orchestra.run.smithi139.stdout:15 Sep 20:20:31 ntpd[18492]: ntpd: time slew -0.001099 s 2024-09-15T20:20:31.182 INFO:teuthology.orchestra.run.smithi139.stdout:ntpd: time slew -0.001099s 2024-09-15T20:20:31.183 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:31 ntpd[18492]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-15T20:20:31.183 INFO:teuthology.orchestra.run.smithi139.stderr:15 Sep 20:20:31 ntpd[18492]: can't open /var/log/ntpstats/loopstats.20240915: Permission denied 2024-09-15T20:20:31.237 INFO:teuthology.orchestra.run.smithi139.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T20:20:31.238 INFO:teuthology.orchestra.run.smithi139.stdout:============================================================================== 2024-09-15T20:20:31.238 INFO:teuthology.orchestra.run.smithi139.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-15T20:20:31.238 INFO:teuthology.orchestra.run.smithi139.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-15T20:20:31.238 INFO:teuthology.orchestra.run.smithi139.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-15T20:20:31.238 INFO:teuthology.orchestra.run.smithi139.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-15T20:20:31.238 INFO:teuthology.run_tasks:Running task install... 2024-09-15T20:20:31.248 DEBUG:teuthology.task.install:project ceph 2024-09-15T20:20:31.254 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}} 2024-09-15T20:20:31.254 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'} 2024-09-15T20:20:31.254 INFO:teuthology.task.install:Using flavor: default 2024-09-15T20:20:31.280 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T20:20:31.295 INFO:teuthology.task.install:extra packages: [] 2024-09-15T20:20:31.295 DEBUG:teuthology.orchestra.run.smithi139:> sudo apt-key list | grep Ceph 2024-09-15T20:20:31.361 INFO:teuthology.orchestra.run.smithi139.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-15T20:20:31.392 INFO:teuthology.orchestra.run.smithi139.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-15T20:20:31.392 INFO:teuthology.orchestra.run.smithi139.stdout:uid [ unknown] Ceph.com (release key) 2024-09-15T20:20:31.392 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-15T20:20:31.393 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T20:20:31.556 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default/ 2024-09-15T20:20:31.556 INFO:teuthology.task.install.deb:Package version is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-15T20:20:31.681 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-15T20:20:31.681 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-15T20:20:31.699 DEBUG:teuthology.orchestra.run.smithi139:> sudo apt-get update 2024-09-15T20:20:31.868 INFO:teuthology.orchestra.run.smithi139.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-15T20:20:31.875 INFO:teuthology.orchestra.run.smithi139.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-15T20:20:31.892 INFO:teuthology.orchestra.run.smithi139.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-15T20:20:31.921 INFO:teuthology.orchestra.run.smithi139.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy InRelease 2024-09-15T20:20:31.942 INFO:teuthology.orchestra.run.smithi139.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-15T20:20:31.964 INFO:teuthology.orchestra.run.smithi139.stdout:Ign:6 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-15T20:20:31.985 INFO:teuthology.orchestra.run.smithi139.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-15T20:20:31.991 INFO:teuthology.orchestra.run.smithi139.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-15T20:20:32.012 INFO:teuthology.orchestra.run.smithi139.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,427 B] 2024-09-15T20:20:32.210 INFO:teuthology.orchestra.run.smithi139.stdout:Fetched 29.2 kB in 0s (88.6 kB/s) 2024-09-15T20:20:33.238 INFO:teuthology.orchestra.run.smithi139.stdout:Reading package lists... 2024-09-15T20:20:33.259 DEBUG:teuthology.orchestra.run.smithi139:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4930-gfc3e0bff-1jammy cephadm=19.3.0-4930-gfc3e0bff-1jammy ceph-mds=19.3.0-4930-gfc3e0bff-1jammy ceph-mgr=19.3.0-4930-gfc3e0bff-1jammy ceph-common=19.3.0-4930-gfc3e0bff-1jammy ceph-fuse=19.3.0-4930-gfc3e0bff-1jammy ceph-test=19.3.0-4930-gfc3e0bff-1jammy ceph-volume=19.3.0-4930-gfc3e0bff-1jammy radosgw=19.3.0-4930-gfc3e0bff-1jammy python3-rados=19.3.0-4930-gfc3e0bff-1jammy python3-rgw=19.3.0-4930-gfc3e0bff-1jammy python3-cephfs=19.3.0-4930-gfc3e0bff-1jammy python3-rbd=19.3.0-4930-gfc3e0bff-1jammy libcephfs2=19.3.0-4930-gfc3e0bff-1jammy libcephfs-dev=19.3.0-4930-gfc3e0bff-1jammy librados2=19.3.0-4930-gfc3e0bff-1jammy librbd1=19.3.0-4930-gfc3e0bff-1jammy rbd-fuse=19.3.0-4930-gfc3e0bff-1jammy 2024-09-15T20:20:33.327 INFO:teuthology.orchestra.run.smithi139.stdout:Reading package lists... 2024-09-15T20:20:33.543 INFO:teuthology.orchestra.run.smithi139.stdout:Building dependency tree... 2024-09-15T20:20:33.543 INFO:teuthology.orchestra.run.smithi139.stdout:Reading state information... 2024-09-15T20:20:33.752 INFO:teuthology.orchestra.run.smithi139.stdout:The following packages were automatically installed and are no longer required: 2024-09-15T20:20:33.752 INFO:teuthology.orchestra.run.smithi139.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-15T20:20:33.752 INFO:teuthology.orchestra.run.smithi139.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-15T20:20:33.752 INFO:teuthology.orchestra.run.smithi139.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-15T20:20:33.752 INFO:teuthology.orchestra.run.smithi139.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-15T20:20:33.753 INFO:teuthology.orchestra.run.smithi139.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-15T20:20:33.753 INFO:teuthology.orchestra.run.smithi139.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-15T20:20:33.753 INFO:teuthology.orchestra.run.smithi139.stdout:The following additional packages will be installed: 2024-09-15T20:20:33.753 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-15T20:20:33.755 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-15T20:20:33.755 INFO:teuthology.orchestra.run.smithi139.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-15T20:20:33.755 INFO:teuthology.orchestra.run.smithi139.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-15T20:20:33.755 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-15T20:20:33.755 INFO:teuthology.orchestra.run.smithi139.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-15T20:20:33.755 INFO:teuthology.orchestra.run.smithi139.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-15T20:20:33.755 INFO:teuthology.orchestra.run.smithi139.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-15T20:20:33.755 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-15T20:20:33.755 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-15T20:20:33.755 INFO:teuthology.orchestra.run.smithi139.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-15T20:20:33.755 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-15T20:20:33.756 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-15T20:20:33.756 INFO:teuthology.orchestra.run.smithi139.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-15T20:20:33.756 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-15T20:20:33.756 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-15T20:20:33.756 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-15T20:20:33.756 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-15T20:20:33.756 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-15T20:20:33.756 INFO:teuthology.orchestra.run.smithi139.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-15T20:20:33.756 INFO:teuthology.orchestra.run.smithi139.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-15T20:20:33.756 INFO:teuthology.orchestra.run.smithi139.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-15T20:20:33.756 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-15T20:20:33.756 INFO:teuthology.orchestra.run.smithi139.stdout: xmlstarlet zip 2024-09-15T20:20:33.757 INFO:teuthology.orchestra.run.smithi139.stdout:Suggested packages: 2024-09-15T20:20:33.757 INFO:teuthology.orchestra.run.smithi139.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-15T20:20:33.757 INFO:teuthology.orchestra.run.smithi139.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-15T20:20:33.757 INFO:teuthology.orchestra.run.smithi139.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-15T20:20:33.757 INFO:teuthology.orchestra.run.smithi139.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-15T20:20:33.758 INFO:teuthology.orchestra.run.smithi139.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-15T20:20:33.758 INFO:teuthology.orchestra.run.smithi139.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-15T20:20:33.758 INFO:teuthology.orchestra.run.smithi139.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-15T20:20:33.758 INFO:teuthology.orchestra.run.smithi139.stdout:Recommended packages: 2024-09-15T20:20:33.758 INFO:teuthology.orchestra.run.smithi139.stdout: btrfs-tools 2024-09-15T20:20:33.860 INFO:teuthology.orchestra.run.smithi139.stdout:The following NEW packages will be installed: 2024-09-15T20:20:33.861 INFO:teuthology.orchestra.run.smithi139.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-15T20:20:33.861 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-15T20:20:33.861 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-15T20:20:33.862 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-15T20:20:33.862 INFO:teuthology.orchestra.run.smithi139.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-15T20:20:33.863 INFO:teuthology.orchestra.run.smithi139.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-15T20:20:33.863 INFO:teuthology.orchestra.run.smithi139.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-15T20:20:33.863 INFO:teuthology.orchestra.run.smithi139.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-15T20:20:33.863 INFO:teuthology.orchestra.run.smithi139.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-15T20:20:33.863 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-15T20:20:33.863 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-15T20:20:33.863 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-15T20:20:33.863 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-15T20:20:33.863 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-15T20:20:33.863 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-15T20:20:33.863 INFO:teuthology.orchestra.run.smithi139.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-15T20:20:33.863 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-15T20:20:33.863 INFO:teuthology.orchestra.run.smithi139.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-15T20:20:33.863 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-15T20:20:33.864 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-15T20:20:33.864 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-15T20:20:33.864 INFO:teuthology.orchestra.run.smithi139.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-15T20:20:33.864 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-15T20:20:33.864 INFO:teuthology.orchestra.run.smithi139.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-15T20:20:33.864 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-15T20:20:33.864 INFO:teuthology.orchestra.run.smithi139.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-15T20:20:34.005 INFO:teuthology.orchestra.run.smithi139.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-15T20:20:34.005 INFO:teuthology.orchestra.run.smithi139.stdout:Need to get 244 MB of archives. 2024-09-15T20:20:34.005 INFO:teuthology.orchestra.run.smithi139.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-15T20:20:34.005 INFO:teuthology.orchestra.run.smithi139.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,633 kB] 2024-09-15T20:20:34.051 INFO:teuthology.orchestra.run.smithi139.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-15T20:20:34.205 INFO:teuthology.orchestra.run.smithi139.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,334 kB] 2024-09-15T20:20:34.250 INFO:teuthology.orchestra.run.smithi139.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4930-gfc3e0bff-1jammy [721 kB] 2024-09-15T20:20:34.260 INFO:teuthology.orchestra.run.smithi139.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4930-gfc3e0bff-1jammy [356 kB] 2024-09-15T20:20:34.266 INFO:teuthology.orchestra.run.smithi139.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4930-gfc3e0bff-1jammy [32.8 kB] 2024-09-15T20:20:34.266 INFO:teuthology.orchestra.run.smithi139.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4930-gfc3e0bff-1jammy [180 kB] 2024-09-15T20:20:34.269 INFO:teuthology.orchestra.run.smithi139.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4930-gfc3e0bff-1jammy [71.3 kB] 2024-09-15T20:20:34.270 INFO:teuthology.orchestra.run.smithi139.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4930-gfc3e0bff-1jammy [338 kB] 2024-09-15T20:20:34.277 INFO:teuthology.orchestra.run.smithi139.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4930-gfc3e0bff-1jammy [7,966 kB] 2024-09-15T20:20:34.413 INFO:teuthology.orchestra.run.smithi139.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4930-gfc3e0bff-1jammy [112 kB] 2024-09-15T20:20:34.413 INFO:teuthology.orchestra.run.smithi139.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4930-gfc3e0bff-1jammy [473 kB] 2024-09-15T20:20:34.416 INFO:teuthology.orchestra.run.smithi139.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4930-gfc3e0bff-1jammy [28.2 MB] 2024-09-15T20:20:34.470 INFO:teuthology.orchestra.run.smithi139.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-15T20:20:34.483 INFO:teuthology.orchestra.run.smithi139.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-15T20:20:34.568 INFO:teuthology.orchestra.run.smithi139.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-15T20:20:34.791 INFO:teuthology.orchestra.run.smithi139.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-15T20:20:34.802 INFO:teuthology.orchestra.run.smithi139.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-15T20:20:34.832 INFO:teuthology.orchestra.run.smithi139.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-15T20:20:34.841 INFO:teuthology.orchestra.run.smithi139.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-15T20:20:34.843 INFO:teuthology.orchestra.run.smithi139.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-15T20:20:34.844 INFO:teuthology.orchestra.run.smithi139.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-15T20:20:34.845 INFO:teuthology.orchestra.run.smithi139.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-15T20:20:34.863 INFO:teuthology.orchestra.run.smithi139.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-15T20:20:34.867 INFO:teuthology.orchestra.run.smithi139.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-15T20:20:34.873 INFO:teuthology.orchestra.run.smithi139.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-15T20:20:34.953 INFO:teuthology.orchestra.run.smithi139.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-15T20:20:35.012 INFO:teuthology.orchestra.run.smithi139.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-15T20:20:35.012 INFO:teuthology.orchestra.run.smithi139.stdout:Get:29 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4930-gfc3e0bff-1jammy [5,206 kB] 2024-09-15T20:20:35.017 INFO:teuthology.orchestra.run.smithi139.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-15T20:20:35.020 INFO:teuthology.orchestra.run.smithi139.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-15T20:20:35.023 INFO:teuthology.orchestra.run.smithi139.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-15T20:20:35.024 INFO:teuthology.orchestra.run.smithi139.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-15T20:20:35.025 INFO:teuthology.orchestra.run.smithi139.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-15T20:20:35.027 INFO:teuthology.orchestra.run.smithi139.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-15T20:20:35.028 INFO:teuthology.orchestra.run.smithi139.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-15T20:20:35.036 INFO:teuthology.orchestra.run.smithi139.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-15T20:20:35.074 INFO:teuthology.orchestra.run.smithi139.stdout:Get:38 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4930-gfc3e0bff-1jammy [251 kB] 2024-09-15T20:20:35.077 INFO:teuthology.orchestra.run.smithi139.stdout:Get:39 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [124 kB] 2024-09-15T20:20:35.078 INFO:teuthology.orchestra.run.smithi139.stdout:Get:40 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4930-gfc3e0bff-1jammy [1,561 kB] 2024-09-15T20:20:35.091 INFO:teuthology.orchestra.run.smithi139.stdout:Get:41 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4930-gfc3e0bff-1jammy [6,309 kB] 2024-09-15T20:20:35.111 INFO:teuthology.orchestra.run.smithi139.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-15T20:20:35.111 INFO:teuthology.orchestra.run.smithi139.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-15T20:20:35.112 INFO:teuthology.orchestra.run.smithi139.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-15T20:20:35.113 INFO:teuthology.orchestra.run.smithi139.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-15T20:20:35.118 INFO:teuthology.orchestra.run.smithi139.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-15T20:20:35.119 INFO:teuthology.orchestra.run.smithi139.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-15T20:20:35.120 INFO:teuthology.orchestra.run.smithi139.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-15T20:20:35.121 INFO:teuthology.orchestra.run.smithi139.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-15T20:20:35.123 INFO:teuthology.orchestra.run.smithi139.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-15T20:20:35.189 INFO:teuthology.orchestra.run.smithi139.stdout:Get:51 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4930-gfc3e0bff-1jammy [23.1 MB] 2024-09-15T20:20:35.199 INFO:teuthology.orchestra.run.smithi139.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-15T20:20:35.199 INFO:teuthology.orchestra.run.smithi139.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-15T20:20:35.202 INFO:teuthology.orchestra.run.smithi139.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-15T20:20:35.281 INFO:teuthology.orchestra.run.smithi139.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-15T20:20:35.281 INFO:teuthology.orchestra.run.smithi139.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-15T20:20:35.293 INFO:teuthology.orchestra.run.smithi139.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-15T20:20:35.293 INFO:teuthology.orchestra.run.smithi139.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-15T20:20:35.294 INFO:teuthology.orchestra.run.smithi139.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-15T20:20:35.295 INFO:teuthology.orchestra.run.smithi139.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-15T20:20:35.298 INFO:teuthology.orchestra.run.smithi139.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-15T20:20:35.362 INFO:teuthology.orchestra.run.smithi139.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-15T20:20:35.369 INFO:teuthology.orchestra.run.smithi139.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-15T20:20:35.369 INFO:teuthology.orchestra.run.smithi139.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-15T20:20:35.444 INFO:teuthology.orchestra.run.smithi139.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-15T20:20:35.451 INFO:teuthology.orchestra.run.smithi139.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-15T20:20:35.451 INFO:teuthology.orchestra.run.smithi139.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-15T20:20:35.451 INFO:teuthology.orchestra.run.smithi139.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-15T20:20:35.455 INFO:teuthology.orchestra.run.smithi139.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-15T20:20:35.673 INFO:teuthology.orchestra.run.smithi139.stdout:Get:70 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [14.1 kB] 2024-09-15T20:20:35.673 INFO:teuthology.orchestra.run.smithi139.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-15T20:20:35.673 INFO:teuthology.orchestra.run.smithi139.stdout:Get:72 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [880 kB] 2024-09-15T20:20:35.674 INFO:teuthology.orchestra.run.smithi139.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-15T20:20:35.680 INFO:teuthology.orchestra.run.smithi139.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-15T20:20:35.681 INFO:teuthology.orchestra.run.smithi139.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-15T20:20:35.681 INFO:teuthology.orchestra.run.smithi139.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-15T20:20:35.685 INFO:teuthology.orchestra.run.smithi139.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4930-gfc3e0bff-1jammy [2,452 kB] 2024-09-15T20:20:35.719 INFO:teuthology.orchestra.run.smithi139.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4930-gfc3e0bff-1jammy [813 kB] 2024-09-15T20:20:35.724 INFO:teuthology.orchestra.run.smithi139.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-15T20:20:35.729 INFO:teuthology.orchestra.run.smithi139.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-15T20:20:35.729 INFO:teuthology.orchestra.run.smithi139.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4930-gfc3e0bff-1jammy [160 kB] 2024-09-15T20:20:35.729 INFO:teuthology.orchestra.run.smithi139.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-15T20:20:35.730 INFO:teuthology.orchestra.run.smithi139.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4930-gfc3e0bff-1jammy [61.4 MB] 2024-09-15T20:20:35.779 INFO:teuthology.orchestra.run.smithi139.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-15T20:20:35.779 INFO:teuthology.orchestra.run.smithi139.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-15T20:20:35.841 INFO:teuthology.orchestra.run.smithi139.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-15T20:20:35.841 INFO:teuthology.orchestra.run.smithi139.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-15T20:20:35.842 INFO:teuthology.orchestra.run.smithi139.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-15T20:20:35.843 INFO:teuthology.orchestra.run.smithi139.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-15T20:20:35.844 INFO:teuthology.orchestra.run.smithi139.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-15T20:20:35.845 INFO:teuthology.orchestra.run.smithi139.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-15T20:20:35.853 INFO:teuthology.orchestra.run.smithi139.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-15T20:20:35.921 INFO:teuthology.orchestra.run.smithi139.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-15T20:20:35.924 INFO:teuthology.orchestra.run.smithi139.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-15T20:20:35.926 INFO:teuthology.orchestra.run.smithi139.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-15T20:20:36.003 INFO:teuthology.orchestra.run.smithi139.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-15T20:20:36.009 INFO:teuthology.orchestra.run.smithi139.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-15T20:20:36.013 INFO:teuthology.orchestra.run.smithi139.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-15T20:20:36.014 INFO:teuthology.orchestra.run.smithi139.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-15T20:20:36.025 INFO:teuthology.orchestra.run.smithi139.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-15T20:20:36.026 INFO:teuthology.orchestra.run.smithi139.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-15T20:20:36.039 INFO:teuthology.orchestra.run.smithi139.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-15T20:20:36.084 INFO:teuthology.orchestra.run.smithi139.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-15T20:20:36.085 INFO:teuthology.orchestra.run.smithi139.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-15T20:20:36.086 INFO:teuthology.orchestra.run.smithi139.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-15T20:20:36.169 INFO:teuthology.orchestra.run.smithi139.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-15T20:20:36.169 INFO:teuthology.orchestra.run.smithi139.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-15T20:20:36.188 INFO:teuthology.orchestra.run.smithi139.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-15T20:20:36.188 INFO:teuthology.orchestra.run.smithi139.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-15T20:20:36.188 INFO:teuthology.orchestra.run.smithi139.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-15T20:20:36.193 INFO:teuthology.orchestra.run.smithi139.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-15T20:20:36.194 INFO:teuthology.orchestra.run.smithi139.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-15T20:20:37.005 INFO:teuthology.orchestra.run.smithi139.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4930-gfc3e0bff-1jammy [8,625 kB] 2024-09-15T20:20:37.114 INFO:teuthology.orchestra.run.smithi139.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4930-gfc3e0bff-1jammy [14.2 kB] 2024-09-15T20:20:37.114 INFO:teuthology.orchestra.run.smithi139.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4930-gfc3e0bff-1jammy [48.5 MB] 2024-09-15T20:20:38.156 INFO:teuthology.orchestra.run.smithi139.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4930-gfc3e0bff-1jammy [130 kB] 2024-09-15T20:20:38.158 INFO:teuthology.orchestra.run.smithi139.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4930-gfc3e0bff-1jammy [39.0 kB] 2024-09-15T20:20:38.159 INFO:teuthology.orchestra.run.smithi139.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4930-gfc3e0bff-1jammy [15.8 MB] 2024-09-15T20:20:38.460 INFO:teuthology.orchestra.run.smithi139.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [91.2 kB] 2024-09-15T20:20:39.041 INFO:teuthology.orchestra.run.smithi139.stdout:Fetched 244 MB in 5s (53.1 MB/s) 2024-09-15T20:20:39.210 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-15T20:20:39.311 INFO:teuthology.orchestra.run.smithi139.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-15T20:20:39.316 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-15T20:20:39.400 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-15T20:20:39.746 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-15T20:20:39.774 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-15T20:20:39.813 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-15T20:20:40.082 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-15T20:20:40.108 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-15T20:20:40.140 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-15T20:20:40.577 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-15T20:20:40.603 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-15T20:20:40.651 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T20:20:41.030 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-15T20:20:41.056 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-15T20:20:41.096 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T20:20:41.374 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-15T20:20:41.400 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-15T20:20:41.432 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T20:20:41.794 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-15T20:20:41.821 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-15T20:20:41.861 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-15T20:20:42.197 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package librados2. 2024-09-15T20:20:42.223 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../007-librados2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:20:42.263 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:20:42.683 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libnbd0. 2024-09-15T20:20:42.709 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-15T20:20:42.741 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-15T20:20:43.044 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package librbd1. 2024-09-15T20:20:43.071 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../009-librbd1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:20:43.110 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:20:43.555 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libcephfs2. 2024-09-15T20:20:43.582 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:20:43.621 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:20:43.874 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-rados. 2024-09-15T20:20:43.889 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:20:43.925 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:20:44.210 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-15T20:20:44.236 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:20:44.278 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:20:44.512 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-cephfs. 2024-09-15T20:20:44.528 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:20:44.562 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:20:44.840 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-15T20:20:44.866 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:20:44.906 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:20:45.236 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-15T20:20:45.261 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-15T20:20:45.301 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-15T20:20:45.621 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-prettytable. 2024-09-15T20:20:45.649 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-15T20:20:45.687 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-15T20:20:45.923 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-rbd. 2024-09-15T20:20:45.939 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:20:45.974 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:20:46.284 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-15T20:20:46.311 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-15T20:20:46.359 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-15T20:20:46.628 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-15T20:20:46.645 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-15T20:20:46.686 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-15T20:20:46.984 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-15T20:20:47.008 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-15T20:20:47.039 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-15T20:20:47.344 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package lua5.1. 2024-09-15T20:20:47.369 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-15T20:20:47.409 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-15T20:20:47.720 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package lua-any. 2024-09-15T20:20:47.747 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-15T20:20:47.787 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-15T20:20:48.039 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package zip. 2024-09-15T20:20:48.063 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-15T20:20:48.098 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking zip (3.0-12build2) ... 2024-09-15T20:20:48.384 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package unzip. 2024-09-15T20:20:48.410 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-15T20:20:48.459 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-15T20:20:48.830 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package luarocks. 2024-09-15T20:20:48.856 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-15T20:20:48.897 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-15T20:20:49.279 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package librgw2. 2024-09-15T20:20:49.305 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../026-librgw2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:20:49.345 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:20:49.923 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-rgw. 2024-09-15T20:20:49.938 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:20:49.972 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:20:50.742 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-15T20:20:50.769 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-15T20:20:50.959 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-15T20:20:51.287 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libradosstriper1. 2024-09-15T20:20:51.316 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:20:51.353 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:20:51.707 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-common. 2024-09-15T20:20:51.733 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:20:51.775 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:20:53.253 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-base. 2024-09-15T20:20:53.280 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:20:53.346 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:20:53.899 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-15T20:20:53.926 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-15T20:20:53.965 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-15T20:20:54.262 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-cheroot. 2024-09-15T20:20:54.289 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-15T20:20:54.320 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-15T20:20:54.615 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-15T20:20:54.641 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-15T20:20:54.681 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-15T20:20:54.984 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-15T20:20:55.010 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-15T20:20:55.050 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-15T20:20:55.373 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-15T20:20:55.397 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-15T20:20:55.437 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-15T20:20:55.723 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-tempora. 2024-09-15T20:20:55.749 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-15T20:20:55.781 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-15T20:20:56.093 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-portend. 2024-09-15T20:20:56.119 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-15T20:20:56.159 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-15T20:20:56.470 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-15T20:20:56.497 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-15T20:20:56.537 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-15T20:20:56.823 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-15T20:20:56.849 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-15T20:20:56.889 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-15T20:20:57.268 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-natsort. 2024-09-15T20:20:57.294 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-15T20:20:57.334 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-15T20:20:57.654 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-logutils. 2024-09-15T20:20:57.680 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-15T20:20:57.720 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-15T20:20:57.998 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-15T20:20:58.024 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-15T20:20:58.064 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-15T20:20:58.317 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-mako. 2024-09-15T20:20:58.344 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-15T20:20:58.383 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-15T20:20:58.703 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-15T20:20:58.730 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-15T20:20:58.770 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-15T20:20:59.073 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-15T20:20:59.100 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-15T20:20:59.139 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-15T20:20:59.442 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-webob. 2024-09-15T20:20:59.468 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-15T20:20:59.508 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-15T20:20:59.836 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-waitress. 2024-09-15T20:20:59.865 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-15T20:20:59.906 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-15T20:21:00.183 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-tempita. 2024-09-15T20:21:00.208 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-15T20:21:00.239 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-15T20:21:00.525 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-paste. 2024-09-15T20:21:00.551 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-15T20:21:00.593 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-15T20:21:00.894 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-15T20:21:00.910 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-15T20:21:00.952 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-15T20:21:01.405 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-15T20:21:01.433 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-15T20:21:01.472 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-15T20:21:01.776 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-webtest. 2024-09-15T20:21:01.801 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-15T20:21:01.841 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-15T20:21:02.152 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-pecan. 2024-09-15T20:21:02.179 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-15T20:21:02.219 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-15T20:21:02.582 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-certifi. 2024-09-15T20:21:02.609 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-15T20:21:02.648 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-15T20:21:02.951 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-idna. 2024-09-15T20:21:02.966 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-15T20:21:03.001 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-15T20:21:03.312 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-urllib3. 2024-09-15T20:21:03.339 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-15T20:21:03.381 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-15T20:21:03.695 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-requests. 2024-09-15T20:21:03.717 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-15T20:21:03.757 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-15T20:21:04.068 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-15T20:21:04.094 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-15T20:21:04.134 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-15T20:21:04.429 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-15T20:21:04.456 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:21:04.495 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:04.883 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-15T20:21:04.917 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:21:04.950 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:05.236 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-15T20:21:05.262 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-15T20:21:05.302 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-15T20:21:05.614 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-openssl. 2024-09-15T20:21:05.640 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-15T20:21:05.680 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-15T20:21:06.017 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-mgr. 2024-09-15T20:21:06.043 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:21:06.083 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:06.469 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-mon. 2024-09-15T20:21:06.496 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:21:06.535 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:07.564 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-osd. 2024-09-15T20:21:07.591 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:21:07.965 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:09.026 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph. 2024-09-15T20:21:09.053 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../067-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:21:09.092 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:09.411 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-fuse. 2024-09-15T20:21:09.438 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:21:09.479 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:09.823 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-mds. 2024-09-15T20:21:09.849 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:21:09.891 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:10.292 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package cephadm. 2024-09-15T20:21:10.318 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../070-cephadm_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:21:10.359 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:10.669 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-15T20:21:10.696 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-15T20:21:10.739 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-15T20:21:11.050 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-15T20:21:11.063 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-15T20:21:11.105 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-15T20:21:12.226 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-babel. 2024-09-15T20:21:12.253 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-15T20:21:12.284 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-15T20:21:12.612 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-jinja2. 2024-09-15T20:21:12.640 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-15T20:21:12.679 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-15T20:21:13.032 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-15T20:21:13.060 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:21:13.098 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:13.418 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-15T20:21:13.445 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-15T20:21:13.489 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-15T20:21:13.788 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-routes. 2024-09-15T20:21:13.814 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-15T20:21:13.854 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-15T20:21:14.183 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-15T20:21:14.210 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:21:14.248 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:17.065 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-15T20:21:17.077 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-15T20:21:17.119 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-15T20:21:17.548 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-joblib. 2024-09-15T20:21:17.575 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-15T20:21:17.614 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-15T20:21:17.971 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-15T20:21:17.996 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-15T20:21:18.033 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-15T20:21:18.336 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-sklearn. 2024-09-15T20:21:18.364 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-15T20:21:18.490 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-15T20:21:19.247 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-15T20:21:19.275 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:21:19.313 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:20.258 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-cachetools. 2024-09-15T20:21:20.286 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-15T20:21:20.324 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-15T20:21:20.644 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-15T20:21:20.671 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-15T20:21:20.710 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-15T20:21:21.039 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-15T20:21:21.067 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-15T20:21:21.105 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-15T20:21:21.417 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-rsa. 2024-09-15T20:21:21.444 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-15T20:21:21.483 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-15T20:21:21.753 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-google-auth. 2024-09-15T20:21:21.770 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-15T20:21:21.803 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-15T20:21:22.131 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-15T20:21:22.159 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-15T20:21:22.198 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-15T20:21:22.487 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-websocket. 2024-09-15T20:21:22.500 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-15T20:21:22.534 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-15T20:21:22.804 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-15T20:21:22.832 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-15T20:21:22.870 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-15T20:21:23.484 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-15T20:21:23.501 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:21:23.534 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:23.853 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-15T20:21:23.881 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-15T20:21:23.919 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-15T20:21:24.281 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-15T20:21:24.309 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-15T20:21:24.347 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-15T20:21:24.600 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package jq. 2024-09-15T20:21:24.628 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-15T20:21:24.667 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-15T20:21:24.877 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package socat. 2024-09-15T20:21:24.893 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-15T20:21:24.935 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-15T20:21:25.213 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package xmlstarlet. 2024-09-15T20:21:25.230 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-15T20:21:25.272 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-15T20:21:25.626 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-test. 2024-09-15T20:21:25.642 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:21:25.684 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:28.197 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package ceph-volume. 2024-09-15T20:21:28.224 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:21:28.264 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:28.533 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-15T20:21:28.549 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:21:28.583 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:28.894 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-15T20:21:28.922 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-15T20:21:28.962 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-15T20:21:29.282 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-15T20:21:29.309 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-15T20:21:29.348 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-15T20:21:29.702 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package nvme-cli. 2024-09-15T20:21:29.729 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-15T20:21:29.768 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-15T20:21:30.149 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package pkg-config. 2024-09-15T20:21:30.181 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-15T20:21:30.206 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-15T20:21:30.434 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-15T20:21:30.450 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-15T20:21:30.492 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-15T20:21:30.878 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-15T20:21:30.894 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-15T20:21:30.936 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-15T20:21:31.241 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-pastescript. 2024-09-15T20:21:31.270 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-15T20:21:31.307 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-15T20:21:31.628 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-pluggy. 2024-09-15T20:21:31.656 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-15T20:21:31.694 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-15T20:21:32.022 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-psutil. 2024-09-15T20:21:32.050 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-15T20:21:32.093 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-15T20:21:32.433 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-py. 2024-09-15T20:21:32.461 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-15T20:21:32.507 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-15T20:21:32.836 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-pygments. 2024-09-15T20:21:32.864 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-15T20:21:32.902 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-15T20:21:33.348 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-15T20:21:33.376 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-15T20:21:33.417 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-15T20:21:33.717 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-toml. 2024-09-15T20:21:33.746 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-15T20:21:33.783 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-15T20:21:34.088 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-pytest. 2024-09-15T20:21:34.114 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-15T20:21:34.153 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-15T20:21:34.490 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package python3-simplejson. 2024-09-15T20:21:34.522 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-15T20:21:34.556 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-15T20:21:34.842 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-15T20:21:34.859 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-15T20:21:34.900 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-15T20:21:35.637 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package radosgw. 2024-09-15T20:21:35.665 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../117-radosgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:21:35.703 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:36.566 INFO:teuthology.orchestra.run.smithi139.stdout:Selecting previously unselected package rbd-fuse. 2024-09-15T20:21:36.581 INFO:teuthology.orchestra.run.smithi139.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:21:36.623 INFO:teuthology.orchestra.run.smithi139.stdout:Unpacking rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:36.916 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-15T20:21:37.162 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-15T20:21:37.287 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-15T20:21:37.549 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-15T20:21:37.846 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-15T20:21:38.276 INFO:teuthology.orchestra.run.smithi139.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-15T20:21:38.356 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:38.846 INFO:teuthology.orchestra.run.smithi139.stdout:Adding system user cephadm....done 2024-09-15T20:21:39.003 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-15T20:21:39.279 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-15T20:21:39.538 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-15T20:21:39.673 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-15T20:21:39.899 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-15T20:21:40.142 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-15T20:21:40.242 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-15T20:21:40.544 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-15T20:21:40.913 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-15T20:21:41.172 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-15T20:21:41.449 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-15T20:21:41.716 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-15T20:21:41.942 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-15T20:21:42.194 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-15T20:21:42.453 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:42.831 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-15T20:21:43.240 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-15T20:21:43.481 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-15T20:21:43.607 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-15T20:21:43.724 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-15T20:21:43.975 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-15T20:21:44.435 INFO:teuthology.orchestra.run.smithi139.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-15T20:21:44.464 INFO:teuthology.orchestra.run.smithi139.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-15T20:21:44.528 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-15T20:21:44.654 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-15T20:21:45.014 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-15T20:21:45.139 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-15T20:21:45.404 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-15T20:21:45.658 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-15T20:21:45.959 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up zip (3.0-12build2) ... 2024-09-15T20:21:46.085 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-15T20:21:46.720 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-15T20:21:46.980 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-15T20:21:47.106 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-15T20:21:47.208 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-15T20:21:47.491 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-15T20:21:47.893 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-15T20:21:48.143 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-15T20:21:48.547 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-15T20:21:48.854 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-15T20:21:49.222 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-15T20:21:49.456 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-15T20:21:49.582 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:21:49.884 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-15T20:21:50.205 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-15T20:21:51.332 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-15T20:21:51.583 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T20:21:51.717 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-15T20:21:51.985 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-15T20:21:52.112 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-15T20:21:52.237 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-15T20:21:52.530 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-15T20:21:52.772 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-15T20:21:53.025 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T20:21:53.150 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-15T20:21:53.451 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-15T20:21:53.702 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-15T20:21:53.961 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-15T20:21:54.216 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-15T20:21:54.472 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-15T20:21:54.723 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-15T20:21:54.849 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-15T20:21:55.125 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-15T20:21:55.259 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-15T20:21:55.553 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-15T20:21:55.844 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-15T20:21:56.104 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-15T20:21:56.222 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-15T20:21:56.322 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-15T20:21:56.403 INFO:teuthology.orchestra.run.smithi139.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-15T20:21:56.624 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-15T20:21:56.750 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-15T20:21:57.126 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-15T20:21:57.386 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-15T20:21:57.513 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-15T20:21:57.762 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T20:21:57.889 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-15T20:21:58.183 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-15T20:21:58.499 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-15T20:21:58.946 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-15T20:21:59.076 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-15T20:21:59.343 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-15T20:21:59.689 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-15T20:21:59.931 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-15T20:22:00.316 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-15T20:22:00.592 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-15T20:22:00.936 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-15T20:22:01.062 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:01.172 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:01.273 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-15T20:22:01.609 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:01.754 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:01.877 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-15T20:22:02.165 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:02.272 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:02.372 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:02.587 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T20:22:02.588 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T20:22:03.074 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:03.199 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:03.325 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:03.451 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-15T20:22:04.655 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:04.768 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:04.890 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:05.016 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:05.142 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:05.651 INFO:teuthology.orchestra.run.smithi139.stdout:Adding group ceph....done 2024-09-15T20:22:05.986 INFO:teuthology.orchestra.run.smithi139.stdout:Adding system user ceph....done 2024-09-15T20:22:06.155 INFO:teuthology.orchestra.run.smithi139.stdout:Setting system user ceph properties....done 2024-09-15T20:22:06.168 INFO:teuthology.orchestra.run.smithi139.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-15T20:22:06.305 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-15T20:22:06.651 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-15T20:22:07.134 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:07.260 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:07.728 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T20:22:07.728 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T20:22:08.195 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:08.537 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-15T20:22:09.014 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:09.270 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T20:22:09.270 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T20:22:09.799 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:10.015 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T20:22:10.016 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T20:22:10.501 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:10.765 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T20:22:10.766 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T20:22:11.236 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:11.361 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:11.504 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:11.716 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T20:22:11.717 INFO:teuthology.orchestra.run.smithi139.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T20:22:12.164 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:12.315 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:12.445 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:12.592 INFO:teuthology.orchestra.run.smithi139.stdout:Setting up ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:22:12.945 INFO:teuthology.orchestra.run.smithi139.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-15T20:22:13.045 INFO:teuthology.orchestra.run.smithi139.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-15T20:22:13.181 INFO:teuthology.orchestra.run.smithi139.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-15T20:22:14.357 INFO:teuthology.orchestra.run.smithi139.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-15T20:22:15.011 INFO:teuthology.orchestra.run.smithi139.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-15T20:22:15.016 DEBUG:teuthology.parallel:result is None 2024-09-15T20:22:15.017 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T20:22:15.155 DEBUG:teuthology.orchestra.run.smithi139:> dpkg-query -W -f '${Version}' ceph 2024-09-15T20:22:15.173 INFO:teuthology.orchestra.run.smithi139.stdout:19.3.0-4930-gfc3e0bff-1jammy 2024-09-15T20:22:15.174 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-15T20:22:15.174 INFO:teuthology.task.install:The correct ceph version 19.3.0-4930-gfc3e0bff-1jammy is installed. 2024-09-15T20:22:15.176 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T20:22:15.176 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-15T20:22:15.176 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T20:22:15.229 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T20:22:15.230 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-15T20:22:15.230 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T20:22:15.284 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T20:22:15.336 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T20:22:15.336 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-15T20:22:15.336 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T20:22:15.388 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T20:22:15.440 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T20:22:15.441 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-15T20:22:15.441 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T20:22:15.492 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T20:22:15.549 INFO:teuthology.run_tasks:Running task workunit... 2024-09-15T20:22:15.583 INFO:tasks.workunit:Pulling workunits from ref 7498493279982dfe87f00616198a5967475f6169 2024-09-15T20:22:15.583 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-15T20:22:15.583 INFO:tasks.workunit:timeout=3h 2024-09-15T20:22:15.583 INFO:tasks.workunit:cleanup=True 2024-09-15T20:22:15.584 DEBUG:teuthology.orchestra.run.smithi139:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-15T20:22:15.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:22:15.598 INFO:teuthology.orchestra.run.smithi139.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-15T20:22:15.598 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T20:22:15.646 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-15T20:22:15.646 DEBUG:teuthology.orchestra.run.smithi139:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-15T20:22:15.694 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7498493279982dfe87f00616198a5967475f6169 2024-09-15T20:22:15.746 INFO:tasks.workunit.client.0.smithi139.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-15T20:23:11.068 INFO:tasks.workunit.client.0.smithi139.stderr:Note: switching to '7498493279982dfe87f00616198a5967475f6169'. 2024-09-15T20:23:11.068 INFO:tasks.workunit.client.0.smithi139.stderr: 2024-09-15T20:23:11.068 INFO:tasks.workunit.client.0.smithi139.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-15T20:23:11.069 INFO:tasks.workunit.client.0.smithi139.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-15T20:23:11.069 INFO:tasks.workunit.client.0.smithi139.stderr:state without impacting any branches by switching back to a branch. 2024-09-15T20:23:11.069 INFO:tasks.workunit.client.0.smithi139.stderr: 2024-09-15T20:23:11.069 INFO:tasks.workunit.client.0.smithi139.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-15T20:23:11.069 INFO:tasks.workunit.client.0.smithi139.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-15T20:23:11.069 INFO:tasks.workunit.client.0.smithi139.stderr: 2024-09-15T20:23:11.069 INFO:tasks.workunit.client.0.smithi139.stderr: git switch -c 2024-09-15T20:23:11.069 INFO:tasks.workunit.client.0.smithi139.stderr: 2024-09-15T20:23:11.069 INFO:tasks.workunit.client.0.smithi139.stderr:Or undo this operation with: 2024-09-15T20:23:11.069 INFO:tasks.workunit.client.0.smithi139.stderr: 2024-09-15T20:23:11.069 INFO:tasks.workunit.client.0.smithi139.stderr: git switch - 2024-09-15T20:23:11.069 INFO:tasks.workunit.client.0.smithi139.stderr: 2024-09-15T20:23:11.070 INFO:tasks.workunit.client.0.smithi139.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-15T20:23:11.070 INFO:tasks.workunit.client.0.smithi139.stderr: 2024-09-15T20:23:11.070 INFO:tasks.workunit.client.0.smithi139.stderr:HEAD is now at 74984932799 Merge pull request #59797 from zdover23/wip-doc-2024-09-15-install-install-storage-cluster 2024-09-15T20:23:11.071 DEBUG:teuthology.orchestra.run.smithi139:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-15T20:23:11.118 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2024-09-15T20:23:11.118 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-15T20:23:11.165 INFO:tasks.workunit:Running workunits matching osd on client.0... 2024-09-15T20:23:11.166 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2024-09-15T20:23:11.166 DEBUG:teuthology.orchestra.run.smithi139:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2024-09-15T20:23:11.233 INFO:tasks.workunit.client.0.smithi139.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T20:23:11.263 INFO:tasks.workunit.client.0.smithi139.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T20:23:11.263 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-15T20:23:11.264 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-15T20:23:11.264 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-15T20:23:11.264 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-15T20:23:11.264 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-15T20:23:11.264 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-15T20:23:11.264 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-15T20:23:11.264 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-15T20:23:11.264 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/bad-inc-map 2024-09-15T20:23:11.264 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2024-09-15T20:23:11.264 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2024-09-15T20:23:11.264 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:50359 2024-09-15T20:23:11.264 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:50359 2024-09-15T20:23:11.264 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2024-09-15T20:23:11.265 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2024-09-15T20:23:11.265 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=021294c6-eda9-468e-923a-5847806c2176 --auth-supported=none ' 2024-09-15T20:23:11.265 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:50359 ' 2024-09-15T20:23:11.265 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2024-09-15T20:23:11.265 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2024-09-15T20:23:11.266 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T20:23:11.269 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2024-09-15T20:23:11.269 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2024-09-15T20:23:11.269 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2024-09-15T20:23:11.270 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/bad-inc-map 2024-09-15T20:23:11.270 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/bad-inc-map 2024-09-15T20:23:11.270 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/bad-inc-map 2024-09-15T20:23:11.270 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:23:11.270 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/bad-inc-map KILL 2024-09-15T20:23:11.270 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:23:11.270 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:23:11.270 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:23:11.270 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:23:11.270 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:23:11.270 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:23:11.270 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:23:11.270 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:23:11.271 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:23:11.271 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:23:11.271 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:23:11.272 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:23:11.272 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:23:11.272 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:23:11.272 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:23:11.272 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:23:11.272 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:23:11.273 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:23:11.273 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/bad-inc-map 2024-09-15T20:23:11.274 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:23:11.274 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:23:11.274 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21529 2024-09-15T20:23:11.274 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21529 2024-09-15T20:23:11.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:23:11.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:23:11.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/bad-inc-map 2024-09-15T20:23:11.276 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:23:11.276 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:23:11.276 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21529 2024-09-15T20:23:11.276 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.21529 2024-09-15T20:23:11.278 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:23:11.278 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-15T20:23:11.278 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-15T20:23:11.278 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:23:11.278 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2024-09-15T20:23:11.278 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2024-09-15T20:23:11.278 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2024-09-15T20:23:11.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2024-09-15T20:23:11.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/bad-inc-map 2024-09-15T20:23:11.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T20:23:11.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T20:23:11.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T20:23:11.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/bad-inc-map/a 2024-09-15T20:23:11.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2024-09-15T20:23:11.834 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T20:23:11.834 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:23:11.834 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:23:11.835 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:23:11.835 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:23:11.835 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21529 2024-09-15T20:23:11.835 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21529/$cluster-$name.asok' 2024-09-15T20:23:11.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.21529/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T20:23:12.043 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T20:23:12.045 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T20:23:12.045 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:23:12.045 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:23:12.045 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T20:23:12.045 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T20:23:12.045 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:23:12.045 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:23:12.045 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:23:12.045 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:23:12.045 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:23:12.045 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21529 2024-09-15T20:23:12.045 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21529/ceph-mon.a.asok 2024-09-15T20:23:12.046 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:23:12.046 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21529/ceph-mon.a.asok config get fsid 2024-09-15T20:23:12.151 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T20:23:12.151 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:23:12.151 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:23:12.151 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T20:23:12.151 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:23:12.151 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:23:12.151 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:23:12.152 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T20:23:12.152 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:23:12.152 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:23:12.153 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21529 2024-09-15T20:23:12.153 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21529/ceph-mon.a.asok 2024-09-15T20:23:12.153 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:23:12.153 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21529/ceph-mon.a.asok config get mon_host 2024-09-15T20:23:12.251 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2024-09-15T20:23:12.252 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/bad-inc-map 2024-09-15T20:23:12.252 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T20:23:12.252 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T20:23:12.252 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T20:23:12.252 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/bad-inc-map/x 2024-09-15T20:23:12.252 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T20:23:13.156 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T20:23:13.156 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:23:13.157 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:23:13.157 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:23:13.157 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:23:13.157 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21529 2024-09-15T20:23:13.157 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21529/$cluster-$name.asok' 2024-09-15T20:23:13.157 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:23:13.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.21529/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:23:13.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2024-09-15T20:23:13.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/bad-inc-map 2024-09-15T20:23:13.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:23:13.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T20:23:13.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:23:13.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/bad-inc-map/0 2024-09-15T20:23:13.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=021294c6-eda9-468e-923a-5847806c2176 --auth-supported=none --mon-host=127.0.0.1:50359 ' 2024-09-15T20:23:13.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:23:13.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:23:13.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:23:13.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2024-09-15T20:23:13.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2024-09-15T20:23:13.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:23:13.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:23:13.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-15T20:23:13.302 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:23:13.302 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:23:13.302 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:23:13.302 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:23:13.302 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:23:13.302 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21529 2024-09-15T20:23:13.302 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21529/$cluster-$name.asok' 2024-09-15T20:23:13.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21529/$cluster-$name.asok' 2024-09-15T20:23:13.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:23:13.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:23:13.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:23:13.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-15T20:23:13.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-15T20:23:13.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:23:13.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:23:13.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:23:13.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:23:13.305 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:23:13.305 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:23:13.305 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/bad-inc-map/0 2024-09-15T20:23:13.305 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:23:13.305 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=43b5be85-b873-42cd-ba80-0ff27cc98eba 2024-09-15T20:23:13.305 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 43b5be85-b873-42cd-ba80-0ff27cc98eba' 2024-09-15T20:23:13.305 INFO:tasks.workunit.client.0.smithi139.stdout:add osd0 43b5be85-b873-42cd-ba80-0ff27cc98eba 2024-09-15T20:23:13.305 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:23:13.322 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAxQudmtG08ExAAPb8cpo4qa4fhXwe8ck2LQQ== 2024-09-15T20:23:13.322 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAxQudmtG08ExAAPb8cpo4qa4fhXwe8ck2LQQ=="}' 2024-09-15T20:23:13.322 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 43b5be85-b873-42cd-ba80-0ff27cc98eba -i td/bad-inc-map/0/new.json 2024-09-15T20:23:13.595 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:23:13.609 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/bad-inc-map/0/new.json 2024-09-15T20:23:13.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=021294c6-eda9-468e-923a-5847806c2176 --auth-supported=none --mon-host=127.0.0.1:50359 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21529/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAxQudmtG08ExAAPb8cpo4qa4fhXwe8ck2LQQ== --osd-uuid 43b5be85-b873-42cd-ba80-0ff27cc98eba 2024-09-15T20:23:13.699 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:13.695+0000 7f5c5a0987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:13.702 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:13.699+0000 7f5c5a0987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:13.704 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:13.699+0000 7f5c5a0987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:13.704 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:13.699+0000 7f5c5a0987c0 -1 bdev(0x55c1abe77c00 td/bad-inc-map/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:23:13.704 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:13.699+0000 7f5c5a0987c0 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2024-09-15T20:23:19.825 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/bad-inc-map/0/keyring 2024-09-15T20:23:19.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:23:19.827 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd0 key to auth repository 2024-09-15T20:23:19.827 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T20:23:19.827 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:23:20.065 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T20:23:20.065 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T20:23:20.065 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=021294c6-eda9-468e-923a-5847806c2176 --auth-supported=none --mon-host=127.0.0.1:50359 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21529/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:23:20.066 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:23:20.066 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:23:20.066 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:23:20.096 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:20.091+0000 7f12fc3057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:20.098 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:20.095+0000 7f12fc3057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:20.099 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:20.095+0000 7f12fc3057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:20.287 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T20:23:20.287 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:23:20.287 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:23:20.287 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:23:20.287 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:23:20.288 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:20.288 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:23:20.288 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:23:20.288 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:20.288 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:20.512 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:21.512 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:21.512 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:21.512 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:23:21.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:23:21.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:21.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:21.662 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:21.659+0000 7f12fc3057c0 -1 Falling back to public interface 2024-09-15T20:23:21.743 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:22.744 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:22.744 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:22.744 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:23:22.745 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:23:22.745 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:22.745 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:23.217 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:24.218 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:24.218 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:24.218 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:23:24.219 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:23:24.219 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:24.219 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:24.329 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:24.327+0000 7f12fc3057c0 -1 osd.0 0 log_to_monitors true 2024-09-15T20:23:24.765 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:25.767 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:25.767 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:25.767 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:23:25.767 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:23:25.768 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:25.768 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:26.208 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:27.209 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:27.210 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:27.210 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:23:27.210 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:23:27.210 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:27.210 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:27.664 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:28.665 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:28.665 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:28.666 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:23:28.666 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:23:28.666 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:28.666 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:29.228 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:30.230 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:30.230 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:23:30.230 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:30.230 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:23:30.231 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:30.231 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:30.652 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:31.654 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:31.654 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:31.654 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:23:31.654 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:23:31.655 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:31.655 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:32.184 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:33.185 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:33.185 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:33.186 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:23:33.186 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:23:33.186 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:33.186 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:33.630 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:34.632 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:34.632 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:34.632 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:23:34.633 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:23:34.633 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:34.633 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:35.064 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:36.066 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:36.066 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:36.066 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:23:36.066 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:23:36.067 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:36.067 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:36.817 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:37.819 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:37.819 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:37.819 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:23:37.819 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:23:37.820 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:37.820 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:38.587 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:39.589 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:39.589 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:39.589 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:23:39.590 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:23:39.590 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:39.590 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:40.269 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:41.271 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:41.271 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:41.271 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:23:41.272 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:23:41.272 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:41.272 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:41.735 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:42.738 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:42.738 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:42.738 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:23:42.738 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:23:42.738 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:42.739 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:43.472 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:44.473 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:44.474 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:44.474 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:23:44.474 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:23:44.474 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:44.475 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:45.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:46.222 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:46.222 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:46.222 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:23:46.223 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:23:46.223 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:46.223 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:46.852 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:47.853 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:47.853 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:47.854 INFO:tasks.workunit.client.0.smithi139.stdout:18 2024-09-15T20:23:47.854 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:23:47.854 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:47.854 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:23:48.361 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1420760725,v1:127.0.0.1:6801/1420760725] [v2:127.0.0.1:6802/1420760725,v1:127.0.0.1:6803/1420760725] exists,up 43b5be85-b873-42cd-ba80-0ff27cc98eba 2024-09-15T20:23:48.364 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:23:48.364 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:23:48.365 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:23:48.365 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2024-09-15T20:23:48.365 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/bad-inc-map 2024-09-15T20:23:48.365 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:23:48.365 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T20:23:48.365 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:23:48.365 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/bad-inc-map/1 2024-09-15T20:23:48.365 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=021294c6-eda9-468e-923a-5847806c2176 --auth-supported=none --mon-host=127.0.0.1:50359 ' 2024-09-15T20:23:48.365 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:23:48.365 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:23:48.365 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:23:48.365 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2024-09-15T20:23:48.365 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2024-09-15T20:23:48.366 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:23:48.366 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:23:48.366 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-15T20:23:48.366 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:23:48.366 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:23:48.366 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:23:48.366 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:23:48.366 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:23:48.366 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21529 2024-09-15T20:23:48.366 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21529/$cluster-$name.asok' 2024-09-15T20:23:48.366 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21529/$cluster-$name.asok' 2024-09-15T20:23:48.366 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:23:48.366 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:23:48.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:23:48.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-15T20:23:48.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-15T20:23:48.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:23:48.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:23:48.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:23:48.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:23:48.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:23:48.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:23:48.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/bad-inc-map/1 2024-09-15T20:23:48.368 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:23:48.368 INFO:tasks.workunit.client.0.smithi139.stdout:add osd1 f2618fc8-49af-4689-94af-907cd5df73d6 2024-09-15T20:23:48.368 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f2618fc8-49af-4689-94af-907cd5df73d6 2024-09-15T20:23:48.368 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 f2618fc8-49af-4689-94af-907cd5df73d6' 2024-09-15T20:23:48.368 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:23:48.386 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBUQudmNHUVFxAA82sN8b2ycPWA0b4WDvbrLw== 2024-09-15T20:23:48.386 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBUQudmNHUVFxAA82sN8b2ycPWA0b4WDvbrLw=="}' 2024-09-15T20:23:48.386 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f2618fc8-49af-4689-94af-907cd5df73d6 -i td/bad-inc-map/1/new.json 2024-09-15T20:23:48.939 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:23:48.957 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/bad-inc-map/1/new.json 2024-09-15T20:23:48.958 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=021294c6-eda9-468e-923a-5847806c2176 --auth-supported=none --mon-host=127.0.0.1:50359 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21529/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBUQudmNHUVFxAA82sN8b2ycPWA0b4WDvbrLw== --osd-uuid f2618fc8-49af-4689-94af-907cd5df73d6 2024-09-15T20:23:48.995 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:48.991+0000 7f3963f0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:48.997 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:48.995+0000 7f3963f0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:48.998 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:48.995+0000 7f3963f0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:48.999 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:48.995+0000 7f3963f0d7c0 -1 bdev(0x55a2d8351c00 td/bad-inc-map/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:23:48.999 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:48.995+0000 7f3963f0d7c0 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2024-09-15T20:23:55.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/bad-inc-map/1/keyring 2024-09-15T20:23:55.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:23:55.429 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd1 key to auth repository 2024-09-15T20:23:55.429 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T20:23:55.429 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:23:56.041 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T20:23:56.041 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.1 2024-09-15T20:23:56.041 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:23:56.041 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=021294c6-eda9-468e-923a-5847806c2176 --auth-supported=none --mon-host=127.0.0.1:50359 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21529/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:23:56.042 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:23:56.042 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:23:56.070 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:56.070+0000 7f87edfd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:56.072 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:56.070+0000 7f87edfd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:56.073 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:56.074+0000 7f87edfd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:23:56.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T20:23:56.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:23:56.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:23:56.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:23:56.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:23:56.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:56.524 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:23:56.525 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:23:56.525 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:56.525 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:23:56.950 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:57.295 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:57.294+0000 7f87edfd77c0 -1 Falling back to public interface 2024-09-15T20:23:57.951 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:57.951 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:57.952 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:23:57.952 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:23:57.952 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:57.952 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:23:58.419 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:59.421 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:23:59.421 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:23:59.421 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:23:59.422 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:23:59.422 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:23:59.422 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:23:59.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:23:59.890 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:23:59.890+0000 7f87edfd77c0 -1 osd.1 0 log_to_monitors true 2024-09-15T20:24:00.858 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:00.858 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:00.858 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:24:00.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:24:00.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:00.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:24:01.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:02.318 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:02.318 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:02.318 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:24:02.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:24:02.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:02.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:24:02.841 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:03.842 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:03.843 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:03.843 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:24:03.843 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:24:03.843 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:03.844 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:24:04.306 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:05.307 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:05.308 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:05.308 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:24:05.308 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:24:05.308 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:05.309 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:24:05.862 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:06.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:06.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:06.864 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:24:06.865 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:24:06.865 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:06.865 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:24:07.314 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:08.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:08.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:08.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:24:08.317 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:24:08.317 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:08.317 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:24:08.807 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:09.809 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:09.809 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:09.809 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:24:09.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:24:09.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:09.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:24:10.239 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:11.240 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:11.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:11.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:24:11.241 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:24:11.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:24:11.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:11.781 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:12.782 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:12.782 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:12.782 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:24:12.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:24:12.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:12.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:24:13.350 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:14.352 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:14.352 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:14.352 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:24:14.353 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:24:14.353 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:14.353 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:24:15.306 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:16.308 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:16.308 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:16.308 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:24:16.309 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:24:16.309 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:16.309 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:24:17.499 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:18.501 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:18.501 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:18.501 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:24:18.502 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:24:18.502 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:18.502 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:24:19.431 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:20.433 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:20.433 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:20.433 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:24:20.433 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:24:20.433 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:20.434 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:24:20.963 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:21.964 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:21.965 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:21.965 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:24:21.965 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:24:21.965 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:21.965 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:24:22.828 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:23.829 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:23.830 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:24:23.830 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:23.830 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:24:23.830 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:23.831 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:24:24.304 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/451624868,v1:127.0.0.1:6811/451624868] [v2:127.0.0.1:6812/451624868,v1:127.0.0.1:6813/451624868] exists,up f2618fc8-49af-4689-94af-907cd5df73d6 2024-09-15T20:24:24.307 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:24:24.307 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:24:24.307 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:24:24.307 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2024-09-15T20:24:24.307 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/bad-inc-map 2024-09-15T20:24:24.307 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:24:24.307 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T20:24:24.307 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:24:24.307 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/bad-inc-map/2 2024-09-15T20:24:24.307 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=021294c6-eda9-468e-923a-5847806c2176 --auth-supported=none --mon-host=127.0.0.1:50359 ' 2024-09-15T20:24:24.308 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:24:24.308 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:24:24.308 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:24:24.308 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2024-09-15T20:24:24.308 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2024-09-15T20:24:24.308 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:24:24.308 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:24:24.308 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-15T20:24:24.308 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:24:24.308 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:24:24.308 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:24:24.308 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:24:24.308 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:24:24.309 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21529 2024-09-15T20:24:24.309 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21529/$cluster-$name.asok' 2024-09-15T20:24:24.309 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21529/$cluster-$name.asok' 2024-09-15T20:24:24.309 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:24:24.309 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:24:24.310 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:24:24.310 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-15T20:24:24.310 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-15T20:24:24.310 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:24:24.310 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:24:24.310 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:24:24.310 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:24:24.310 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:24:24.310 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:24:24.310 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/bad-inc-map/2 2024-09-15T20:24:24.311 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:24:24.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c73e3e05-7144-4384-a855-9ba64476d4ce 2024-09-15T20:24:24.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 c73e3e05-7144-4384-a855-9ba64476d4ce' 2024-09-15T20:24:24.311 INFO:tasks.workunit.client.0.smithi139.stdout:add osd2 c73e3e05-7144-4384-a855-9ba64476d4ce 2024-09-15T20:24:24.311 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:24:24.331 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB4Qudm0eGwExAA8T4ORtGqpwoHav6OTOjR8Q== 2024-09-15T20:24:24.331 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB4Qudm0eGwExAA8T4ORtGqpwoHav6OTOjR8Q=="}' 2024-09-15T20:24:24.331 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c73e3e05-7144-4384-a855-9ba64476d4ce -i td/bad-inc-map/2/new.json 2024-09-15T20:24:24.857 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:24:24.874 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/bad-inc-map/2/new.json 2024-09-15T20:24:24.875 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=021294c6-eda9-468e-923a-5847806c2176 --auth-supported=none --mon-host=127.0.0.1:50359 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21529/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB4Qudm0eGwExAA8T4ORtGqpwoHav6OTOjR8Q== --osd-uuid c73e3e05-7144-4384-a855-9ba64476d4ce 2024-09-15T20:24:24.907 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:24:24.904+0000 7f89b73507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:24:24.909 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:24:24.904+0000 7f89b73507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:24:24.910 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:24:24.908+0000 7f89b73507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:24:24.911 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:24:24.908+0000 7f89b73507c0 -1 bdev(0x556c37613c00 td/bad-inc-map/2/block) open stat got: (1) Operation not permitted 2024-09-15T20:24:24.911 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:24:24.908+0000 7f89b73507c0 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2024-09-15T20:24:30.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/bad-inc-map/2/keyring 2024-09-15T20:24:30.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:24:30.990 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd2 key to auth repository 2024-09-15T20:24:30.991 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T20:24:30.991 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:24:31.571 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.2 2024-09-15T20:24:31.571 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T20:24:31.571 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=021294c6-eda9-468e-923a-5847806c2176 --auth-supported=none --mon-host=127.0.0.1:50359 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21529/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:24:31.572 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:24:31.572 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:24:31.572 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:24:31.600 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:24:31.596+0000 7f812c2cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:24:31.603 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:24:31.600+0000 7f812c2cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:24:31.605 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:24:31.600+0000 7f812c2cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:24:31.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T20:24:31.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:24:31.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:24:31.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:24:31.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:24:31.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:31.991 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:24:31.991 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:24:31.991 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:31.991 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:24:32.479 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:33.081 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:24:33.076+0000 7f812c2cf7c0 -1 Falling back to public interface 2024-09-15T20:24:33.481 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:33.481 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:33.481 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:24:33.481 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:24:33.481 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:33.481 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:24:33.924 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:34.926 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:34.926 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:34.926 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:24:34.926 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:24:34.927 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:34.927 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:24:35.341 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:35.942 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:24:35.936+0000 7f812c2cf7c0 -1 osd.2 0 log_to_monitors true 2024-09-15T20:24:36.342 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:36.342 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:36.343 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:24:36.343 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:24:36.343 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:36.343 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:24:36.774 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:37.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:37.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:37.776 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:24:37.777 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:24:37.777 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:37.777 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:24:38.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:39.226 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:39.226 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:39.226 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:24:39.227 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:24:39.227 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:39.227 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:24:39.769 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:40.770 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:40.770 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:40.771 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:24:40.771 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:24:40.771 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:40.771 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:24:41.314 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:42.315 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:42.315 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:42.315 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:24:42.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:24:42.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:42.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:24:42.862 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:43.863 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:43.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:43.864 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:24:43.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:24:43.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:43.865 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:24:44.308 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:45.310 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:45.310 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:45.310 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:24:45.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:24:45.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:45.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:24:45.771 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:46.773 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:46.773 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:46.773 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:24:46.774 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:24:46.774 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:46.774 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:24:47.217 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:48.219 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:48.219 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:48.220 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:24:48.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:24:48.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:48.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:24:48.695 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:49.697 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:49.697 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:49.697 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:24:49.698 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:24:49.698 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:49.698 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:24:50.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:51.130 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:51.131 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:51.131 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:24:51.131 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:24:51.131 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:51.131 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:24:51.941 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:52.942 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:52.943 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:52.943 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:24:52.944 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:24:52.944 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:52.945 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:24:53.669 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:54.671 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:54.671 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:54.672 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:24:54.672 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:24:54.672 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:54.672 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:24:55.410 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:56.411 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:56.411 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:56.412 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:24:56.412 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:24:56.412 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:56.412 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:24:57.402 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:58.404 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:58.404 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:58.404 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:24:58.405 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:24:58.405 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:58.405 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:24:58.916 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:24:59.918 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:24:59.918 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:24:59.918 INFO:tasks.workunit.client.0.smithi139.stdout:18 2024-09-15T20:24:59.919 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:24:59.919 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:24:59.919 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:25:00.390 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2765266446,v1:127.0.0.1:6819/2765266446] [v2:127.0.0.1:6820/2765266446,v1:127.0.0.1:6821/2765266446] exists,up c73e3e05-7144-4384-a855-9ba64476d4ce 2024-09-15T20:25:00.391 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:25:00.391 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:25:00.391 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:25:00.391 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2024-09-15T20:25:00.881 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2024-09-15T20:25:00.881 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 8 2024-09-15T20:25:01.477 INFO:tasks.workunit.client.0.smithi139.stderr:pool 'foo' already exists 2024-09-15T20:25:01.494 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:25:02.496 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2024-09-15T20:25:03.863 INFO:tasks.workunit.client.0.smithi139.stderr:set pool 1 min_size to 1 2024-09-15T20:25:03.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2024-09-15T20:25:04.997 INFO:tasks.workunit.client.0.smithi139.stderr:set pool 1 min_size to 2 2024-09-15T20:25:05.018 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2024-09-15T20:25:10.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2024-09-15T20:25:10.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2024-09-15T20:25:10.021 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:25:10.021 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:25:10.021 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:25:10.021 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:25:10.021 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 10 )) 2024-09-15T20:25:10.021 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:25:10.021 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:25:10.022 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:10.022 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:25:10.510 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 5 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1420760725,v1:127.0.0.1:6801/1420760725] [v2:127.0.0.1:6802/1420760725,v1:127.0.0.1:6803/1420760725] exists,up 43b5be85-b873-42cd-ba80-0ff27cc98eba 2024-09-15T20:25:10.511 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:25:10.511 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:25:10.512 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:25:10.512 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2024-09-15T20:25:10.512 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2024-09-15T20:25:10.512 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:25:10.512 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:25:10.512 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:25:10.512 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:25:10.512 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 10 )) 2024-09-15T20:25:10.512 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:25:10.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:25:10.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:10.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:25:10.976 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 10 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/451624868,v1:127.0.0.1:6811/451624868] [v2:127.0.0.1:6812/451624868,v1:127.0.0.1:6813/451624868] exists,up f2618fc8-49af-4689-94af-907cd5df73d6 2024-09-15T20:25:10.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:25:10.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:25:10.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:25:10.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2024-09-15T20:25:10.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2024-09-15T20:25:10.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:25:10.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:25:10.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:25:10.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:25:10.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 10 )) 2024-09-15T20:25:10.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:25:10.978 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:25:10.979 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:10.979 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:25:11.468 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up in weight 1 up_from 15 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2765266446,v1:127.0.0.1:6819/2765266446] [v2:127.0.0.1:6820/2765266446,v1:127.0.0.1:6821/2765266446] exists,up c73e3e05-7144-4384-a855-9ba64476d4ce 2024-09-15T20:25:11.469 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:25:11.469 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:25:11.469 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:25:11.469 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2024-09-15T20:25:11.472 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:01.315+0000 7f811d273640 -1 osd.2 15 handle_osd_map injecting map crc failure 2024-09-15T20:25:11.472 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:01.675+0000 7f811d273640 -1 osd.2 17 handle_osd_map injecting map crc failure 2024-09-15T20:25:11.472 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:02.187+0000 7f811d273640 -1 osd.2 18 handle_osd_map injecting map crc failure 2024-09-15T20:25:11.472 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:02.895+0000 7f811b26f640 -1 osd.2 19 handle_osd_map injecting map crc failure 2024-09-15T20:25:11.472 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:03.119+0000 7f811d273640 -1 osd.2 20 handle_osd_map injecting map crc failure 2024-09-15T20:25:11.472 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:03.295+0000 7f811b26f640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-09-15T20:25:11.473 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:03.367+0000 7f811b26f640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-09-15T20:25:11.473 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:03.727+0000 7f811b26f640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-09-15T20:25:11.473 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:03.815+0000 7f811b26f640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-09-15T20:25:11.473 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:03.959+0000 7f811d273640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-09-15T20:25:11.473 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:04.343+0000 7f811d273640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-09-15T20:25:11.473 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:04.347+0000 7f811d273640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-09-15T20:25:11.473 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:04.695+0000 7f811d273640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-09-15T20:25:11.473 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:05.215+0000 7f811b26f640 -1 osd.2 26 handle_osd_map injecting map crc failure 2024-09-15T20:25:11.473 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:05.307+0000 7f811b26f640 -1 osd.2 26 handle_osd_map injecting map crc failure 2024-09-15T20:25:11.473 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:05.411+0000 7f811d273640 -1 osd.2 26 handle_osd_map injecting map crc failure 2024-09-15T20:25:11.473 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2024-09-15T20:25:11.476 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:01.319+0000 7f811d273640 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2024-09-15T20:25:11.476 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:01.675+0000 7f811d273640 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2024-09-15T20:25:11.476 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:02.191+0000 7f811d273640 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2024-09-15T20:25:11.477 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:02.895+0000 7f811b26f640 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2024-09-15T20:25:11.477 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:03.119+0000 7f811d273640 10 osd.2 20 handle_osd_map bailing because last < start (20<21) 2024-09-15T20:25:11.477 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:03.295+0000 7f811b26f640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2024-09-15T20:25:11.477 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:03.367+0000 7f811b26f640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2024-09-15T20:25:11.477 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:03.727+0000 7f811b26f640 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2024-09-15T20:25:11.477 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:03.815+0000 7f811b26f640 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2024-09-15T20:25:11.477 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:03.959+0000 7f811d273640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2024-09-15T20:25:11.477 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:04.347+0000 7f811d273640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-09-15T20:25:11.477 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:04.347+0000 7f811d273640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-09-15T20:25:11.477 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:04.695+0000 7f811d273640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-09-15T20:25:11.477 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:05.215+0000 7f811b26f640 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2024-09-15T20:25:11.477 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:05.307+0000 7f811b26f640 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2024-09-15T20:25:11.477 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:25:05.415+0000 7f811d273640 10 osd.2 26 handle_osd_map bailing because last < start (27<28) 2024-09-15T20:25:11.477 INFO:tasks.workunit.client.0.smithi139.stdout:success 2024-09-15T20:25:11.478 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2024-09-15T20:25:11.478 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2024-09-15T20:25:11.478 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=foo 2024-09-15T20:25:11.478 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-15T20:25:12.059 INFO:tasks.workunit.client.0.smithi139.stderr:pool 'foo' does not exist 2024-09-15T20:25:12.076 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2024-09-15T20:25:12.077 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:25:12.077 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:25:12.077 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:25:12.077 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:25:12.078 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:25:17.395 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:25:17.395 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2024-09-15T20:25:17.395 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/bad-inc-map 2024-09-15T20:25:17.395 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:25:17.395 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/bad-inc-map KILL 2024-09-15T20:25:17.396 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:25:17.396 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:25:17.396 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:25:17.396 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:25:17.396 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:25:17.403 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:25:17.404 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:25:17.404 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:25:17.405 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:25:17.406 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:25:17.406 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:25:17.406 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:25:17.407 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:25:17.407 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:25:17.407 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:25:17.407 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:25:17.408 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:25:17.409 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:25:17.409 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/bad-inc-map 2024-09-15T20:25:17.424 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:25:17.425 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:25:17.425 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21529 2024-09-15T20:25:17.425 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21529 2024-09-15T20:25:17.426 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:25:17.426 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:25:17.426 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-15T20:25:17.426 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/bad-inc-map 0 2024-09-15T20:25:17.427 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/bad-inc-map 2024-09-15T20:25:17.427 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-15T20:25:17.427 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/bad-inc-map KILL 2024-09-15T20:25:17.427 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:25:17.427 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:25:17.427 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:25:17.427 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:25:17.427 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:25:17.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:25:17.429 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:25:17.429 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:25:17.429 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:25:17.430 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:25:17.430 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:25:17.431 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:25:17.431 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:25:17.431 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:25:17.432 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:25:17.432 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:25:17.433 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:25:17.434 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T20:25:17.434 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/bad-inc-map 2024-09-15T20:25:17.435 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:25:17.435 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:25:17.435 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21529 2024-09-15T20:25:17.435 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21529 2024-09-15T20:25:17.436 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:25:17.436 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:25:17.436 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-15T20:25:17.437 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T20:25:17.437 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T20:25:17.489 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2024-09-15T20:25:17.489 DEBUG:teuthology.orchestra.run.smithi139:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2024-09-15T20:25:17.541 INFO:tasks.workunit.client.0.smithi139.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T20:25:17.547 INFO:tasks.workunit.client.0.smithi139.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T20:25:17.547 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-15T20:25:17.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-15T20:25:17.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-15T20:25:17.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-15T20:25:17.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-15T20:25:17.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-15T20:25:17.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-15T20:25:17.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-15T20:25:17.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/divergent-priors 2024-09-15T20:25:17.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2024-09-15T20:25:17.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2024-09-15T20:25:17.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2024-09-15T20:25:17.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2024-09-15T20:25:17.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2024-09-15T20:25:17.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2024-09-15T20:25:17.549 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2024-09-15T20:25:17.549 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2024-09-15T20:25:17.549 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2024-09-15T20:25:17.549 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2024-09-15T20:25:17.550 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2024-09-15T20:25:17.551 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2024-09-15T20:25:17.551 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2024-09-15T20:25:17.551 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2024-09-15T20:25:17.551 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2024-09-15T20:25:17.551 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2024-09-15T20:25:17.553 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2024-09-15T20:25:17.553 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2024-09-15T20:25:17.553 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2024-09-15T20:25:17.553 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2024-09-15T20:25:17.553 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2024-09-15T20:25:17.553 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2024-09-15T20:25:17.553 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2024-09-15T20:25:17.553 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2024-09-15T20:25:17.553 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2024-09-15T20:25:17.553 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2024-09-15T20:25:17.554 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none ' 2024-09-15T20:25:17.554 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-09-15T20:25:17.554 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T20:25:17.554 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2024-09-15T20:25:17.554 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:25:17.555 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2024-09-15T20:25:17.555 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T20:25:17.557 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2024-09-15T20:25:17.557 INFO:tasks.workunit.client.0.smithi139.stderr:TEST_divergent_2 2024-09-15T20:25:17.558 INFO:tasks.workunit.client.0.smithi139.stderr:TEST_divergent_3 2024-09-15T20:25:17.558 INFO:tasks.workunit.client.0.smithi139.stderr:TEST_divergent_ec' 2024-09-15T20:25:17.558 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-15T20:25:17.558 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-15T20:25:17.558 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-15T20:25:17.558 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-15T20:25:17.558 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-15T20:25:17.558 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:25:17.558 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-15T20:25:17.558 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:25:17.558 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:25:17.558 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:25:17.559 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:25:17.559 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:25:17.559 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:25:17.560 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:25:17.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:25:17.561 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:25:17.562 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:25:17.562 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:25:17.562 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:25:17.563 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:25:17.563 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:25:17.563 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:25:17.563 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:25:17.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:25:17.565 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:25:17.565 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-15T20:25:17.566 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:25:17.567 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:25:17.567 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:25:17.567 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27419 2024-09-15T20:25:17.567 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:25:17.567 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:25:17.568 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-15T20:25:17.569 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:25:17.569 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:25:17.569 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:25:17.570 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27419 2024-09-15T20:25:17.570 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:25:17.571 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-15T20:25:17.571 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-15T20:25:17.571 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:25:17.572 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-15T20:25:17.572 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2024-09-15T20:25:17.572 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2024-09-15T20:25:17.572 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2024-09-15T20:25:17.572 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2024-09-15T20:25:17.572 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2024-09-15T20:25:17.572 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2024-09-15T20:25:17.573 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2024-09-15T20:25:17.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2024-09-15T20:25:17.582 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:25:17.582 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:25:17.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2024-09-15T20:25:17.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-15T20:25:17.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T20:25:17.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T20:25:17.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T20:25:17.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-15T20:25:17.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-15T20:25:18.065 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T20:25:18.066 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:25:18.066 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:25:18.066 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:25:18.066 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:25:18.067 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:25:18.067 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:25:18.067 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T20:25:18.277 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T20:25:18.278 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T20:25:18.278 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:25:18.278 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:25:18.278 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T20:25:18.279 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T20:25:18.279 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:25:18.279 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:25:18.279 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:25:18.279 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:25:18.279 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:25:18.279 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:25:18.279 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-mon.a.asok 2024-09-15T20:25:18.280 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:25:18.280 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27419/ceph-mon.a.asok config get fsid 2024-09-15T20:25:18.396 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T20:25:18.396 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:25:18.396 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:25:18.396 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T20:25:18.397 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T20:25:18.397 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:25:18.397 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:25:18.397 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:25:18.397 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:25:18.397 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:25:18.397 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:25:18.397 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-mon.a.asok 2024-09-15T20:25:18.398 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:25:18.398 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27419/ceph-mon.a.asok config get mon_host 2024-09-15T20:25:18.508 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2024-09-15T20:25:18.508 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-15T20:25:18.508 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T20:25:18.508 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T20:25:18.508 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T20:25:18.509 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-15T20:25:18.509 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T20:25:19.432 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T20:25:19.432 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:25:19.432 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:25:19.433 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:25:19.433 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:25:19.433 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:25:19.433 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:25:19.434 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:25:19.435 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:25:19.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-15T20:25:19.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2024-09-15T20:25:19.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-15T20:25:19.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:25:19.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T20:25:19.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:25:19.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-15T20:25:19.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:25:19.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:25:19.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:25:19.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:25:19.514 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-15T20:25:19.514 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-15T20:25:19.514 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:25:19.514 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:25:19.514 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:25:19.514 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:25:19.514 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:25:19.514 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:25:19.514 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:25:19.514 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:25:19.514 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:25:19.514 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:25:19.514 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:25:19.515 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:25:19.515 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:25:19.515 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:25:19.515 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:25:19.515 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:25:19.515 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:25:19.515 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:25:19.515 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:25:19.515 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:25:19.515 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:25:19.515 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:25:19.515 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-15T20:25:19.516 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:25:19.516 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=cdf05e2d-6e34-45ea-89d8-b5b29cbb7f44 2024-09-15T20:25:19.516 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 cdf05e2d-6e34-45ea-89d8-b5b29cbb7f44' 2024-09-15T20:25:19.516 INFO:tasks.workunit.client.0.smithi139.stdout:add osd0 cdf05e2d-6e34-45ea-89d8-b5b29cbb7f44 2024-09-15T20:25:19.516 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:25:19.539 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCvQudm2acFIBAAtLn4j8kbVBi1423QSh0ZUw== 2024-09-15T20:25:19.539 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCvQudm2acFIBAAtLn4j8kbVBi1423QSh0ZUw=="}' 2024-09-15T20:25:19.539 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new cdf05e2d-6e34-45ea-89d8-b5b29cbb7f44 -i td/divergent-priors/0/new.json 2024-09-15T20:25:19.838 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:25:19.853 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-15T20:25:19.854 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCvQudm2acFIBAAtLn4j8kbVBi1423QSh0ZUw== --osd-uuid cdf05e2d-6e34-45ea-89d8-b5b29cbb7f44 2024-09-15T20:25:19.903 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:25:19.902+0000 7f2c21bad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:25:19.905 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:25:19.902+0000 7f2c21bad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:25:19.906 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:25:19.902+0000 7f2c21bad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:25:19.907 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:25:19.906+0000 7f2c21bad7c0 -1 bdev(0x55768d48fc00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:25:19.907 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:25:19.906+0000 7f2c21bad7c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-15T20:25:25.910 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-15T20:25:25.910 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:25:25.911 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd0 key to auth repository 2024-09-15T20:25:25.911 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T20:25:25.911 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:25:26.556 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T20:25:26.556 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T20:25:26.557 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:25:26.557 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:25:26.557 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:25:26.557 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:25:26.598 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:25:26.594+0000 7fcca6ed87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:25:26.600 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:25:26.598+0000 7fcca6ed87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:25:26.601 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:25:26.598+0000 7fcca6ed87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:25:27.025 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T20:25:27.025 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:25:27.025 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:25:27.025 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:25:27.025 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:25:27.025 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:25:27.025 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:25:27.026 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:25:27.026 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:25:27.026 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:27.491 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:25:27.834 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:25:27.830+0000 7fcca6ed87c0 -1 Falling back to public interface 2024-09-15T20:25:28.493 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:25:28.493 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:25:28.493 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:25:28.493 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:25:28.493 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:28.493 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:25:28.919 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:25:29.921 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:25:29.921 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:25:29.921 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:25:29.922 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:25:29.922 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:29.922 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:25:30.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:25:30.657 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:25:30.654+0000 7fcca6ed87c0 -1 osd.0 0 log_to_monitors true 2024-09-15T20:25:31.368 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:25:31.368 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:25:31.369 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:25:31.369 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:25:31.369 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:31.369 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:25:31.953 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:25:32.955 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:25:32.955 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:25:32.955 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:25:32.956 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:25:32.956 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:32.956 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:25:33.518 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:25:34.520 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:25:34.520 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:25:34.521 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:25:34.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:25:34.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:34.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:25:34.998 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:25:36.000 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:25:36.000 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:25:36.000 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:25:36.000 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:25:36.001 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:36.001 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:25:36.506 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:25:37.508 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:25:37.509 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:25:37.509 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:25:37.509 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:25:37.509 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:37.509 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:25:37.946 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:25:38.948 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:25:38.948 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:25:38.948 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:25:38.949 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:25:38.949 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:38.949 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:25:39.481 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:25:40.482 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:25:40.482 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:25:40.482 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:25:40.483 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:25:40.483 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:40.483 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:25:40.949 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:25:41.951 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:25:41.952 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:25:41.952 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:25:41.952 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:25:41.952 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:41.952 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:25:42.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:25:43.540 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:25:43.540 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:25:43.540 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:25:43.541 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:25:43.541 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:43.541 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:25:44.445 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:25:45.446 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:25:45.446 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:25:45.446 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:25:45.447 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:25:45.447 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:45.447 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:25:46.179 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:25:47.181 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:25:47.181 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:25:47.181 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:25:47.182 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:25:47.182 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:47.182 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:25:48.371 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:25:49.372 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:25:49.373 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:25:49.373 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:25:49.373 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:25:49.374 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:49.374 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:25:50.390 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:25:51.392 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:25:51.392 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:25:51.392 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:25:51.393 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:25:51.393 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:51.393 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:25:51.936 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:25:52.938 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:25:52.938 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:25:52.939 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:25:52.939 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:25:52.939 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:25:52.939 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:25:53.411 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/512977897,v1:127.0.0.1:6803/512977897] [v2:127.0.0.1:6804/512977897,v1:127.0.0.1:6805/512977897] exists,up cdf05e2d-6e34-45ea-89d8-b5b29cbb7f44 2024-09-15T20:25:53.414 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:25:53.414 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:25:53.414 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:25:53.414 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-15T20:25:53.414 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2024-09-15T20:25:53.414 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-15T20:25:53.414 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:25:53.414 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T20:25:53.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:25:53.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-15T20:25:53.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:25:53.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:25:53.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:25:53.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:25:53.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T20:25:53.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T20:25:53.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:25:53.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:25:53.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:25:53.415 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:25:53.415 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:25:53.416 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:25:53.416 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:25:53.416 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:25:53.416 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:25:53.416 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:25:53.416 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:25:53.416 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:25:53.416 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:25:53.416 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:25:53.416 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:25:53.416 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:25:53.416 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:25:53.416 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:25:53.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:25:53.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:25:53.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:25:53.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:25:53.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-15T20:25:53.417 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:25:53.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=95c45dfa-5b44-4aaf-9fc1-ebb69bbe1237 2024-09-15T20:25:53.418 INFO:tasks.workunit.client.0.smithi139.stdout:add osd1 95c45dfa-5b44-4aaf-9fc1-ebb69bbe1237 2024-09-15T20:25:53.418 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 95c45dfa-5b44-4aaf-9fc1-ebb69bbe1237' 2024-09-15T20:25:53.418 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:25:53.438 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDRQudmUxsKGhAA0mMSG5AoblclzY86zrvZ7w== 2024-09-15T20:25:53.439 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDRQudmUxsKGhAA0mMSG5AoblclzY86zrvZ7w=="}' 2024-09-15T20:25:53.439 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 95c45dfa-5b44-4aaf-9fc1-ebb69bbe1237 -i td/divergent-priors/1/new.json 2024-09-15T20:25:53.950 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:25:53.967 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-15T20:25:53.969 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRQudmUxsKGhAA0mMSG5AoblclzY86zrvZ7w== --osd-uuid 95c45dfa-5b44-4aaf-9fc1-ebb69bbe1237 2024-09-15T20:25:54.011 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:25:54.009+0000 7f0c040e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:25:54.013 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:25:54.009+0000 7f0c040e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:25:54.015 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:25:54.009+0000 7f0c040e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:25:54.015 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:25:54.013+0000 7f0c040e07c0 -1 bdev(0x559d05e49c00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:25:54.015 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:25:54.013+0000 7f0c040e07c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-15T20:25:59.720 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-15T20:25:59.720 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:25:59.721 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd1 key to auth repository 2024-09-15T20:25:59.721 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T20:25:59.721 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:26:00.324 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.1 2024-09-15T20:26:00.324 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T20:26:00.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:26:00.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:26:00.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:26:00.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:26:00.366 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:26:00.361+0000 7f79a9d2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:26:00.368 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:26:00.365+0000 7f79a9d2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:26:00.369 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:26:00.365+0000 7f79a9d2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:26:00.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T20:26:00.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:26:00.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:26:00.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:26:00.785 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:26:00.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:26:00.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:00.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:26:00.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:00.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:26:01.272 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:01.886 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:26:01.881+0000 7f79a9d2d7c0 -1 Falling back to public interface 2024-09-15T20:26:02.274 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:02.274 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:02.274 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:26:02.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:26:02.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:02.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:26:02.717 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:03.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:03.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:03.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:26:03.719 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:26:03.720 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:03.720 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:26:04.181 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:04.842 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:26:04.837+0000 7f79a9d2d7c0 -1 osd.1 0 log_to_monitors true 2024-09-15T20:26:05.183 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:05.183 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:05.183 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:26:05.184 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:26:05.184 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:05.184 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:26:05.646 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:06.647 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:06.647 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:06.648 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:26:06.648 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:26:06.648 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:06.648 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:26:07.250 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:08.252 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:08.252 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:08.252 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:26:08.253 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:26:08.253 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:08.253 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:26:08.781 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:09.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:09.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:09.783 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:26:09.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:26:09.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:09.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:26:10.235 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:11.236 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:11.237 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:26:11.237 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:11.237 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:26:11.237 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:11.237 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:26:11.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:12.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:12.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:12.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:26:12.786 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:26:12.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:12.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:26:13.386 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:14.388 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:14.388 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:14.388 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:26:14.389 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:26:14.389 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:14.389 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:26:15.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:16.023 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:16.023 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:16.023 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:26:16.023 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:26:16.024 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:16.024 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:26:16.485 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:17.487 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:17.487 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:17.487 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:26:17.487 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:26:17.488 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:17.488 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:26:18.388 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:19.389 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:19.389 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:19.390 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:26:19.390 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:26:19.390 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:19.390 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:26:20.331 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:21.332 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:21.333 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:21.333 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:26:21.333 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:26:21.333 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:21.333 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:26:22.321 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:23.323 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:23.323 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:23.323 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:26:23.324 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:26:23.324 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:23.324 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:26:24.272 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:25.274 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:25.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:25.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:26:25.275 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:26:25.276 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:25.276 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:26:26.562 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:27.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:27.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:27.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:26:27.564 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:26:27.565 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:27.565 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:26:28.051 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:29.053 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:29.053 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:29.053 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:26:29.053 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:26:29.054 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:29.054 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:26:29.657 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3778108260,v1:127.0.0.1:6811/3778108260] [v2:127.0.0.1:6812/3778108260,v1:127.0.0.1:6813/3778108260] exists,up 95c45dfa-5b44-4aaf-9fc1-ebb69bbe1237 2024-09-15T20:26:29.661 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:26:29.661 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:26:29.661 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:26:29.661 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-15T20:26:29.661 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2024-09-15T20:26:29.661 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-15T20:26:29.661 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:26:29.661 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T20:26:29.661 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:26:29.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-15T20:26:29.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:26:29.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:26:29.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:26:29.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:26:29.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-15T20:26:29.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-15T20:26:29.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:26:29.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:26:29.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:26:29.662 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:26:29.662 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:26:29.663 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:26:29.663 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:26:29.663 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:26:29.663 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:26:29.663 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:26:29.663 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:26:29.663 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:26:29.663 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:26:29.663 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:26:29.663 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:26:29.663 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:26:29.663 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:26:29.663 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:26:29.664 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:26:29.664 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:26:29.664 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:26:29.664 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:26:29.664 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-15T20:26:29.664 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:26:29.664 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9d0447d3-39ad-48fb-8e36-2d8f2ba5c93b 2024-09-15T20:26:29.665 INFO:tasks.workunit.client.0.smithi139.stdout:add osd2 9d0447d3-39ad-48fb-8e36-2d8f2ba5c93b 2024-09-15T20:26:29.665 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 9d0447d3-39ad-48fb-8e36-2d8f2ba5c93b' 2024-09-15T20:26:29.665 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:26:29.694 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD1Qudmrns7KRAAdqL+jqeelyfDHz3dbm9g8Q== 2024-09-15T20:26:29.694 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD1Qudmrns7KRAAdqL+jqeelyfDHz3dbm9g8Q=="}' 2024-09-15T20:26:29.694 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9d0447d3-39ad-48fb-8e36-2d8f2ba5c93b -i td/divergent-priors/2/new.json 2024-09-15T20:26:30.352 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:26:30.368 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-15T20:26:30.370 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD1Qudmrns7KRAAdqL+jqeelyfDHz3dbm9g8Q== --osd-uuid 9d0447d3-39ad-48fb-8e36-2d8f2ba5c93b 2024-09-15T20:26:30.412 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:26:30.408+0000 7f05d72047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:26:30.455 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:26:30.452+0000 7f05d72047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:26:30.457 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:26:30.452+0000 7f05d72047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:26:30.458 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:26:30.452+0000 7f05d72047c0 -1 bdev(0x5618347c5c00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-15T20:26:30.458 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:26:30.452+0000 7f05d72047c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-15T20:26:36.300 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-15T20:26:36.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:26:36.302 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd2 key to auth repository 2024-09-15T20:26:36.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T20:26:36.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:26:36.877 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.2 2024-09-15T20:26:36.877 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T20:26:36.877 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:26:36.877 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:26:36.877 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:26:36.877 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:26:36.917 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:26:36.912+0000 7fcb0c0627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:26:36.919 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:26:36.916+0000 7fcb0c0627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:26:36.921 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:26:36.916+0000 7fcb0c0627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:26:37.356 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T20:26:37.356 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:26:37.356 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:26:37.356 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:26:37.356 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:26:37.356 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:37.357 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:26:37.357 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:26:37.357 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:37.357 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:26:37.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:38.430 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:26:38.424+0000 7fcb0c0627c0 -1 Falling back to public interface 2024-09-15T20:26:38.789 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:38.789 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:38.790 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:26:38.790 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:26:38.790 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:38.790 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:26:39.282 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:40.284 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:40.284 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:40.284 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:26:40.285 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:26:40.285 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:40.285 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:26:40.736 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:41.219 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:26:41.212+0000 7fcb0c0627c0 -1 osd.2 0 log_to_monitors true 2024-09-15T20:26:41.737 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:41.738 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:41.738 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:26:41.738 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:26:41.738 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:41.738 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:26:42.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:43.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:43.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:43.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:26:43.200 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:26:43.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:43.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:26:43.645 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:44.646 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:44.647 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:44.647 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:26:44.647 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:26:44.647 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:44.647 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:26:45.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:46.197 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:46.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:46.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:26:46.198 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:26:46.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:46.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:26:46.666 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:47.668 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:47.668 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:47.668 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:26:47.669 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:26:47.669 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:47.669 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:26:48.127 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:49.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:49.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:49.129 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:26:49.130 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:26:49.130 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:49.130 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:26:49.645 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:50.646 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:50.646 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:50.647 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:26:50.647 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:26:50.647 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:50.647 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:26:51.318 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:52.320 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:52.320 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:52.320 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:26:52.321 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:26:52.321 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:52.321 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:26:52.898 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:53.899 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:53.899 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:53.900 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:26:53.900 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:26:53.900 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:53.900 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:26:54.747 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:55.749 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:55.749 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:55.749 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:26:55.750 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:26:55.750 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:55.750 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:26:56.541 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:57.542 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:57.542 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:57.543 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:26:57.543 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:26:57.543 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:57.543 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:26:58.388 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:26:59.390 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:26:59.390 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:26:59.390 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:26:59.391 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:26:59.391 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:26:59.391 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:27:00.126 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:01.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:01.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:01.128 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:27:01.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:27:01.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:01.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:27:01.871 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:02.873 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:02.873 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:02.873 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:27:02.874 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:27:02.874 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:02.874 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:27:03.844 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:04.845 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:04.846 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:27:04.846 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:04.846 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:27:04.846 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:04.847 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:27:05.368 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3447816141,v1:127.0.0.1:6819/3447816141] [v2:127.0.0.1:6820/3447816141,v1:127.0.0.1:6821/3447816141] exists,up 9d0447d3-39ad-48fb-8e36-2d8f2ba5c93b 2024-09-15T20:27:05.369 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:27:05.369 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:27:05.369 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:27:05.369 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2024-09-15T20:27:06.059 INFO:tasks.workunit.client.0.smithi139.stderr:noout is set 2024-09-15T20:27:06.082 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2024-09-15T20:27:06.761 INFO:tasks.workunit.client.0.smithi139.stderr:noin is set 2024-09-15T20:27:06.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2024-09-15T20:27:07.606 INFO:tasks.workunit.client.0.smithi139.stderr:nodown is set 2024-09-15T20:27:07.625 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2024-09-15T20:27:07.625 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-15T20:27:08.368 INFO:tasks.workunit.client.0.smithi139.stderr:pool 'test' already exists 2024-09-15T20:27:08.392 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:27:09.394 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2024-09-15T20:27:10.066 INFO:tasks.workunit.client.0.smithi139.stderr:set pool 1 size to 3 2024-09-15T20:27:10.086 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2024-09-15T20:27:10.929 INFO:tasks.workunit.client.0.smithi139.stderr:set pool 1 min_size to 2 2024-09-15T20:27:10.952 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2024-09-15T20:27:10.953 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:27:10.954 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:27:11.438 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:27:11.439 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:27:11.439 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:27:11.439 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:27:11.439 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:27:11.439 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:27:11.609 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836503 2024-09-15T20:27:11.609 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836503 2024-09-15T20:27:11.609 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503' 2024-09-15T20:27:11.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:27:11.610 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:27:11.778 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-15T20:27:11.778 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-15T20:27:11.778 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672977' 2024-09-15T20:27:11.778 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:27:11.778 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:27:11.928 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-15T20:27:11.928 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-15T20:27:11.928 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672977 2-64424509451' 2024-09-15T20:27:11.928 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:27:11.928 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836503 2024-09-15T20:27:11.928 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:27:11.930 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:27:11.931 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836503 2024-09-15T20:27:11.931 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:27:11.931 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836503 2024-09-15T20:27:11.932 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 21474836503 2024-09-15T20:27:11.932 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-15T20:27:11.932 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:27:12.402 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-15T20:27:12.402 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:27:12.403 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-15T20:27:12.404 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:27:12.404 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:27:12.406 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-15T20:27:12.406 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:27:12.407 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.1 seq 42949672977 2024-09-15T20:27:12.407 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-15T20:27:12.407 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-15T20:27:12.408 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:27:12.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-15T20:27:12.860 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:27:12.861 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-15T20:27:12.861 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:27:12.862 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:27:12.863 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-15T20:27:12.863 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:27:12.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-15T20:27:12.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-15T20:27:12.864 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 64424509451 2024-09-15T20:27:12.865 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:27:13.352 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-15T20:27:13.353 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2024-09-15T20:27:13.353 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:27:13.354 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:27:13.354 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:27:13.354 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:27:13.355 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:27:13.355 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:27:13.355 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:27:13.357 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:27:13.357 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:27:13.454 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:27:13.454 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:27:13.455 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:27:13.455 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:27:13.455 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:27:13.455 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:27:13.919 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:27:13.919 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:27:13.919 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:27:13.919 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:27:13.919 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:27:13.919 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:27:14.102 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836506 2024-09-15T20:27:14.102 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836506 2024-09-15T20:27:14.102 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506' 2024-09-15T20:27:14.102 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:27:14.102 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:27:14.285 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672979 2024-09-15T20:27:14.285 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672979 2024-09-15T20:27:14.285 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672979' 2024-09-15T20:27:14.285 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:27:14.285 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:27:14.445 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509453 2024-09-15T20:27:14.445 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509453 2024-09-15T20:27:14.446 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672979 2-64424509453' 2024-09-15T20:27:14.446 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:27:14.446 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:27:14.446 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836506 2024-09-15T20:27:14.447 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:27:14.448 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836506 2024-09-15T20:27:14.448 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:27:14.449 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836506 2024-09-15T20:27:14.449 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 21474836506 2024-09-15T20:27:14.450 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-15T20:27:14.450 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:27:14.878 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-15T20:27:14.878 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:27:14.879 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672979 2024-09-15T20:27:14.879 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:27:14.880 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:27:14.881 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:27:14.882 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672979 2024-09-15T20:27:14.882 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672979 2024-09-15T20:27:14.882 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-15T20:27:14.883 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.1 seq 42949672979 2024-09-15T20:27:14.883 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:27:15.346 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672979 2024-09-15T20:27:15.346 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:27:16.347 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:27:16.348 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:27:16.790 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672980 -lt 42949672979 2024-09-15T20:27:16.791 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:27:16.791 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509453 2024-09-15T20:27:16.792 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:27:16.793 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:27:16.794 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509453 2024-09-15T20:27:16.794 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:27:16.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509453 2024-09-15T20:27:16.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-15T20:27:16.796 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 64424509453 2024-09-15T20:27:16.796 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:27:17.281 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509454 -lt 64424509453 2024-09-15T20:27:17.282 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:27:17.282 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:27:17.282 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:27:17.866 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:27:17.867 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:27:17.868 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:27:17.868 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:27:17.868 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:27:17.869 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:27:17.869 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:27:17.869 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:27:18.295 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:27:18.296 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:27:18.297 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:27:18.297 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:27:18.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:27:18.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:27:18.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:27:18.860 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2024-09-15T20:27:18.860 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2024-09-15T20:27:19.257 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:27:19.273 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2024-09-15T20:27:19.273 INFO:tasks.workunit.client.0.smithi139.stdout:primary and soon to be divergent is 1 2024-09-15T20:27:19.274 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2024-09-15T20:27:19.274 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2024-09-15T20:27:19.684 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:27:19.684 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T20:27:09.498030+0000 0'0 29:24 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:27:08.062076+0000 0'0 2024-09-15T20:27:08.062076+0000 0 0 periodic scrub scheduled @ 2024-09-17T00:52:29.842631+0000 0 0 2024-09-15T20:27:19.684 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:27:19.685 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:27:19.685 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:27:19.701 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2024-09-15T20:27:19.701 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-15T20:27:19.701 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2024-09-15T20:27:19.701 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2024-09-15T20:27:19.701 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-15T20:27:19.701 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2024-09-15T20:27:19.701 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2024-09-15T20:27:19.701 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-15T20:27:19.701 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2024-09-15T20:27:19.701 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2024-09-15T20:27:19.702 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2024-09-15T20:27:19.702 INFO:tasks.workunit.client.0.smithi139.stdout:writing initial objects 2024-09-15T20:27:19.702 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2024-09-15T20:27:19.702 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:19.702 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2024-09-15T20:27:19.880 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:19.880 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2024-09-15T20:27:20.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:20.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2024-09-15T20:27:20.390 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:20.391 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2024-09-15T20:27:20.541 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:20.541 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2024-09-15T20:27:20.701 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:20.702 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2024-09-15T20:27:20.851 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:20.851 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2024-09-15T20:27:21.001 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:21.001 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2024-09-15T20:27:21.144 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:21.144 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2024-09-15T20:27:21.295 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:21.295 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2024-09-15T20:27:21.437 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:21.437 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2024-09-15T20:27:21.588 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:21.588 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2024-09-15T20:27:21.731 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:21.731 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2024-09-15T20:27:21.873 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:21.873 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2024-09-15T20:27:22.032 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:22.032 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2024-09-15T20:27:22.166 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:22.166 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2024-09-15T20:27:22.317 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:22.317 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2024-09-15T20:27:22.468 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:22.468 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2024-09-15T20:27:22.628 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:22.628 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2024-09-15T20:27:22.796 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:22.796 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2024-09-15T20:27:22.945 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:22.946 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2024-09-15T20:27:23.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:23.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2024-09-15T20:27:23.247 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:23.247 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2024-09-15T20:27:23.390 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:23.390 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2024-09-15T20:27:23.532 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:23.532 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2024-09-15T20:27:23.705 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:23.705 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2024-09-15T20:27:23.850 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:23.850 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2024-09-15T20:27:23.992 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:23.993 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2024-09-15T20:27:24.143 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:24.144 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2024-09-15T20:27:24.294 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:24.294 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2024-09-15T20:27:24.436 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:24.437 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2024-09-15T20:27:24.587 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:24.587 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2024-09-15T20:27:24.756 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:24.756 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2024-09-15T20:27:24.898 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:24.898 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2024-09-15T20:27:25.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:25.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2024-09-15T20:27:25.183 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:25.183 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2024-09-15T20:27:25.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:25.326 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2024-09-15T20:27:25.476 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:25.476 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2024-09-15T20:27:25.627 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:25.627 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2024-09-15T20:27:25.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:25.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2024-09-15T20:27:25.936 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:25.937 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2024-09-15T20:27:26.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:26.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2024-09-15T20:27:26.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:26.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2024-09-15T20:27:26.406 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:26.406 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2024-09-15T20:27:26.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:26.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2024-09-15T20:27:26.725 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:26.725 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2024-09-15T20:27:26.867 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:26.867 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2024-09-15T20:27:27.068 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:27.068 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2024-09-15T20:27:27.218 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:27.218 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2024-09-15T20:27:27.361 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:27.361 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2024-09-15T20:27:27.504 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:27.504 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2024-09-15T20:27:27.681 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:27.681 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2024-09-15T20:27:27.840 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:27.841 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2024-09-15T20:27:27.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:27.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2024-09-15T20:27:28.141 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:28.141 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2024-09-15T20:27:28.300 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:28.300 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2024-09-15T20:27:28.451 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:28.451 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2024-09-15T20:27:28.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:28.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2024-09-15T20:27:28.819 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:28.820 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2024-09-15T20:27:28.979 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:28.979 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2024-09-15T20:27:29.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:29.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2024-09-15T20:27:29.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:29.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2024-09-15T20:27:29.424 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:29.424 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2024-09-15T20:27:29.590 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:29.590 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2024-09-15T20:27:29.741 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:29.741 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2024-09-15T20:27:29.883 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:29.883 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2024-09-15T20:27:30.034 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:30.034 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2024-09-15T20:27:30.185 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:30.185 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2024-09-15T20:27:30.337 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:30.337 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2024-09-15T20:27:30.495 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:30.495 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2024-09-15T20:27:30.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:30.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2024-09-15T20:27:30.831 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:30.831 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2024-09-15T20:27:30.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:30.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2024-09-15T20:27:31.148 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:31.149 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2024-09-15T20:27:31.291 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:31.291 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2024-09-15T20:27:31.467 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:31.467 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2024-09-15T20:27:31.618 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:31.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2024-09-15T20:27:31.768 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:31.769 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2024-09-15T20:27:31.927 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:31.927 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2024-09-15T20:27:32.087 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:32.087 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2024-09-15T20:27:32.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:32.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2024-09-15T20:27:32.448 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:32.449 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2024-09-15T20:27:32.615 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:32.615 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2024-09-15T20:27:32.791 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:32.791 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2024-09-15T20:27:32.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:32.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2024-09-15T20:27:33.083 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:33.084 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2024-09-15T20:27:33.235 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:33.235 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2024-09-15T20:27:33.376 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:33.377 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2024-09-15T20:27:33.527 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:33.528 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2024-09-15T20:27:33.677 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:33.678 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2024-09-15T20:27:33.829 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:33.829 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2024-09-15T20:27:33.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:33.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2024-09-15T20:27:34.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:34.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2024-09-15T20:27:34.307 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:34.307 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2024-09-15T20:27:34.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:34.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2024-09-15T20:27:34.591 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:34.592 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2024-09-15T20:27:34.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:34.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2024-09-15T20:27:34.935 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:34.935 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2024-09-15T20:27:35.094 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:35.094 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2024-09-15T20:27:35.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T20:27:35.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-09-15T20:27:35.437 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T20:27:35.437 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2024-09-15T20:27:35.437 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:27:35.438 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:27:35.438 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:27:35.438 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T20:27:35.439 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:27:35.439 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:27:35.439 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:27:35.439 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:27:35.439 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:27:35.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T20:27:35.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:27:35.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:27:35.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:27:35.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:27:35.513 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:27:36.006 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:27:36.006 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:27:36.006 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:27:36.006 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:27:36.006 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:27:36.007 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:27:36.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836512 2024-09-15T20:27:36.170 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836512 2024-09-15T20:27:36.170 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512' 2024-09-15T20:27:36.170 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:27:36.170 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:27:36.334 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672986 2024-09-15T20:27:36.334 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672986 2024-09-15T20:27:36.334 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672986' 2024-09-15T20:27:36.334 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:27:36.334 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:27:36.499 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509460 2024-09-15T20:27:36.499 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509460 2024-09-15T20:27:36.499 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672986 2-64424509460' 2024-09-15T20:27:36.499 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:27:36.499 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836512 2024-09-15T20:27:36.499 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:27:36.500 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:27:36.501 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836512 2024-09-15T20:27:36.501 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:27:36.502 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836512 2024-09-15T20:27:36.503 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 21474836512 2024-09-15T20:27:36.503 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-15T20:27:36.503 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:27:36.935 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836512 2024-09-15T20:27:36.935 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:27:36.936 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672986 2024-09-15T20:27:36.936 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:27:36.937 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:27:36.938 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672986 2024-09-15T20:27:36.938 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:27:36.939 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672986 2024-09-15T20:27:36.939 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-09-15T20:27:36.940 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.1 seq 42949672986 2024-09-15T20:27:36.940 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:27:37.416 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672984 -lt 42949672986 2024-09-15T20:27:37.416 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:27:38.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:27:38.418 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:27:38.892 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672986 -lt 42949672986 2024-09-15T20:27:38.893 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:27:38.894 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509460 2024-09-15T20:27:38.894 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:27:38.895 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:27:38.896 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509460 2024-09-15T20:27:38.897 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:27:38.898 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 64424509460 2024-09-15T20:27:38.899 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509460 2024-09-15T20:27:38.899 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-15T20:27:38.899 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:27:39.369 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509460 -lt 64424509460 2024-09-15T20:27:39.370 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:27:39.370 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:27:39.370 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:27:39.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:27:39.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:27:39.934 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:27:39.935 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:27:39.935 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:27:39.935 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:27:39.935 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:27:39.935 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:27:40.350 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:27:40.351 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:27:40.351 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:27:40.351 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:27:40.919 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:27:40.920 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:27:40.920 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:27:40.920 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2024-09-15T20:27:40.920 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-15T20:27:40.920 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-15T20:27:40.921 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:27:40.921 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-15T20:27:41.426 INFO:tasks.workunit.client.0.smithi139.stdout:blackholing osds 0 2 2024-09-15T20:27:41.426 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2024-09-15T20:27:41.426 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2024-09-15T20:27:41.426 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2024-09-15T20:27:41.821 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:27:41.821 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-09-15T20:27:09.498030+0000 29'100 29:124 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:27:08.062076+0000 0'0 2024-09-15T20:27:08.062076+0000 0 0 periodic scrub scheduled @ 2024-09-17T00:52:29.842631+0000 0 0 2024-09-15T20:27:41.821 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:27:41.822 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:27:41.822 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:27:41.837 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-09-15T20:27:41.837 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2024-09-15T20:27:41.837 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T20:27:41.837 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T20:27:41.838 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:27:41.838 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:27:41.838 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:27:41.839 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.0.asok 2024-09-15T20:27:41.839 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-09-15T20:27:41.839 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-15T20:27:41.933 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:27:41.933 INFO:tasks.workunit.client.0.smithi139.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-15T20:27:41.933 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:27:41.944 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-09-15T20:27:41.945 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2024-09-15T20:27:41.945 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-15T20:27:41.946 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-15T20:27:41.946 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:27:41.946 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:27:41.946 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:27:41.946 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.2.asok 2024-09-15T20:27:41.947 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-09-15T20:27:41.947 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-15T20:27:42.041 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:27:42.041 INFO:tasks.workunit.client.0.smithi139.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-15T20:27:42.041 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:27:42.053 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2024-09-15T20:27:42.055 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2024-09-15T20:27:42.056 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2024-09-15T20:27:42.056 INFO:tasks.workunit.client.0.smithi139.stdout:writing divergent object 2024-09-15T20:27:42.056 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2024-09-15T20:27:42.057 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-09-15T20:27:42.057 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2024-09-15T20:27:42.057 INFO:tasks.workunit.client.0.smithi139.stdout:create missing divergent object 2024-09-15T20:27:42.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2024-09-15T20:27:42.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-15T20:27:42.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-15T20:27:42.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-15T20:27:42.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-15T20:27:42.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=test 2024-09-15T20:27:42.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-15T20:27:42.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=existing_99 2024-09-15T20:27:42.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-15T20:27:42.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/divergent-priors 2024-09-15T20:27:42.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-15T20:27:42.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-15T20:27:42.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-15T20:27:42.062 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds test existing_99 2024-09-15T20:27:42.062 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=test 2024-09-15T20:27:42.062 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=existing_99 2024-09-15T20:27:42.062 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map test existing_99 2024-09-15T20:27:42.062 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-15T20:27:42.535 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-15T20:27:42.535 INFO:tasks.workunit.client.0.smithi139.stderr:0 2024-09-15T20:27:42.535 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:27:42.535 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2 2024-09-15T20:27:42.535 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0' '2') 2024-09-15T20:27:42.535 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-15T20:27:42.536 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-15T20:27:42.536 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-15T20:27:42.536 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-15T20:27:42.536 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/divergent-priors/1/type 2024-09-15T20:27:42.537 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-15T20:27:42.537 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T20:27:42.537 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-15T20:27:42.537 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-15T20:27:42.537 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T20:27:42.537 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-15T20:27:42.538 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-15T20:27:42.538 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-15T20:27:42.538 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T20:27:42.538 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T20:27:42.538 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:27:42.538 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:27:42.538 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:27:42.539 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.1.asok 2024-09-15T20:27:42.539 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.27419/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T20:27:42.654 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-15T20:27:42.654 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-15T20:27:42.654 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-15T20:27:42.655 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-15T20:27:42.655 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T20:27:42.655 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T20:27:42.656 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:27:42.656 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:27:42.656 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:27:42.656 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.1.asok 2024-09-15T20:27:42.656 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-15T20:27:42.656 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.1.asok injectdataerr test existing_99 2024-09-15T20:27:42.773 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2024-09-15T20:27:42.773 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2024-09-15T20:27:52.774 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2024-09-15T20:27:52.787 INFO:tasks.workunit.client.0.smithi139.stderr:rados: no process found 2024-09-15T20:27:52.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2024-09-15T20:27:52.788 INFO:tasks.workunit.client.0.smithi139.stdout:killing all the osds 2024-09-15T20:27:52.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2024-09-15T20:27:53.199 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:27:53.199 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 active+clean 2024-09-15T20:27:42.981070+0000 29'101 29:133 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:27:08.062076+0000 0'0 2024-09-15T20:27:08.062076+0000 0 0 periodic scrub scheduled @ 2024-09-17T00:52:29.842631+0000 0 0 2024-09-15T20:27:53.199 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:27:53.199 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:27:53.199 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:27:53.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2024-09-15T20:27:53.215 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:27:53.215 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:27:53.216 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:27:53.216 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:27:53.216 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:27:53.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:27:53.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-15T20:27:53.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2024-09-15T20:27:53.989 INFO:tasks.workunit.client.0.smithi139.stderr:osd.0 is already down. 2024-09-15T20:27:54.006 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-15T20:27:54.006 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2024-09-15T20:27:54.622 INFO:tasks.workunit.client.0.smithi139.stderr:osd.1 is already down. 2024-09-15T20:27:54.638 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-15T20:27:54.638 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2024-09-15T20:27:55.185 INFO:tasks.workunit.client.0.smithi139.stderr:osd.2 is already down. 2024-09-15T20:27:55.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-09-15T20:27:55.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2024-09-15T20:27:55.757 INFO:tasks.workunit.client.0.smithi139.stderr:osd.0 is already out. 2024-09-15T20:27:55.772 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-09-15T20:27:55.772 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2024-09-15T20:27:56.351 INFO:tasks.workunit.client.0.smithi139.stderr:osd.2 is already out. 2024-09-15T20:27:56.367 INFO:tasks.workunit.client.0.smithi139.stdout:bringing up non_divergent 0 2 2024-09-15T20:27:56.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2024-09-15T20:27:56.368 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2024-09-15T20:27:56.774 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:27:56.774 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 stale+active+clean 2024-09-15T20:27:42.981070+0000 29'101 29:133 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:27:08.062076+0000 0'0 2024-09-15T20:27:08.062076+0000 0 0 periodic scrub scheduled @ 2024-09-17T00:52:29.842631+0000 0 0 2024-09-15T20:27:56.774 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:27:56.774 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:27:56.775 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:27:56.791 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-09-15T20:27:56.791 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2024-09-15T20:27:56.791 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-15T20:27:56.791 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:27:56.791 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:27:56.791 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:27:56.791 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-15T20:27:56.791 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:27:56.792 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:27:56.792 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:27:56.792 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:27:56.792 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-15T20:27:56.792 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-15T20:27:56.792 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:27:56.792 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:27:56.792 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:27:56.792 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:27:56.792 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:27:56.792 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:27:56.794 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:27:56.794 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:27:56.794 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:27:56.794 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:27:56.794 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:27:56.794 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:27:56.794 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:27:56.794 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:27:56.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:27:56.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:27:56.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:27:56.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:27:56.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:27:56.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:27:56.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-15T20:27:56.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:27:56.795 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T20:27:56.796 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:27:56.796 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-15T20:27:56.796 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:27:56.796 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:27:56.796 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:27:56.797 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:27:56.834 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:27:56.827+0000 7f2784aa77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:27:56.836 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:27:56.831+0000 7f2784aa77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:27:56.837 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:27:56.831+0000 7f2784aa77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:27:57.240 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:27:57.240 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:27:57.240 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:27:57.240 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:27:57.240 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:27:57.241 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:27:57.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:57.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:27:57.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:57.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:27:57.682 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:27:58.085 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:27:58.079+0000 7f2784aa77c0 -1 Falling back to public interface 2024-09-15T20:27:58.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:27:58.684 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:27:58.684 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:27:58.684 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:27:58.684 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:27:58.684 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:27:59.116 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:28:00.117 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:28:00.117 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:28:00.118 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:28:00.118 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:28:00.118 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:28:00.118 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:28:00.558 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:28:01.217 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:28:01.211+0000 7f2784aa77c0 -1 osd.0 29 log_to_monitors true 2024-09-15T20:28:01.560 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:28:01.561 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:28:01.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:28:01.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:28:01.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:28:01.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:28:02.068 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/3285624630,v1:127.0.0.1:6803/3285624630] [v2:127.0.0.1:6804/3285624630,v1:127.0.0.1:6805/3285624630] exists,up cdf05e2d-6e34-45ea-89d8-b5b29cbb7f44 2024-09-15T20:28:02.071 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:28:02.071 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:28:02.071 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:28:02.071 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-09-15T20:28:02.071 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2024-09-15T20:28:02.071 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-15T20:28:02.072 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:28:02.072 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T20:28:02.072 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:28:02.072 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-15T20:28:02.072 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:28:02.072 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:28:02.072 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:28:02.072 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:28:02.072 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-15T20:28:02.072 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-15T20:28:02.072 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:28:02.072 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:28:02.072 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:28:02.073 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:28:02.073 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:28:02.073 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:28:02.073 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:28:02.073 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:28:02.073 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:28:02.073 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:28:02.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:28:02.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:28:02.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:28:02.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:28:02.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:28:02.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:28:02.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:28:02.074 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:28:02.074 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:28:02.074 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:28:02.074 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-15T20:28:02.074 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.2 2024-09-15T20:28:02.075 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T20:28:02.075 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:28:02.075 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-15T20:28:02.075 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T20:28:02.075 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:28:02.075 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:28:02.075 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:28:02.108 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:28:02.103+0000 7f03889287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:02.131 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:28:02.127+0000 7f03889287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:02.132 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:28:02.127+0000 7f03889287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:02.579 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T20:28:02.579 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:28:02.579 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:28:02.579 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:28:02.579 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:28:02.579 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:28:02.579 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:28:02.580 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:28:02.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:28:02.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:28:03.062 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:28:03.193 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:28:03.187+0000 7f03889287c0 -1 Falling back to public interface 2024-09-15T20:28:04.064 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:28:04.064 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:28:04.064 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:28:04.065 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:28:04.065 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:28:04.065 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:28:04.545 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:28:05.547 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:28:05.547 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:28:05.547 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:28:05.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:28:05.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:28:05.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:28:06.013 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:28:06.259 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:28:06.255+0000 7f03889287c0 -1 osd.2 29 log_to_monitors true 2024-09-15T20:28:07.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:28:07.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:28:07.015 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:28:07.015 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:28:07.015 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:28:07.015 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:28:07.505 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/919482650,v1:127.0.0.1:6811/919482650] [v2:127.0.0.1:6812/919482650,v1:127.0.0.1:6813/919482650] exists,up 9d0447d3-39ad-48fb-8e36-2d8f2ba5c93b 2024-09-15T20:28:07.505 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:28:07.506 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:28:07.506 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:28:07.506 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-09-15T20:28:07.506 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2024-09-15T20:28:08.073 INFO:tasks.workunit.client.0.smithi139.stderr:osd.0 is already in. 2024-09-15T20:28:08.089 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-09-15T20:28:08.089 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2024-09-15T20:28:08.666 INFO:tasks.workunit.client.0.smithi139.stderr:osd.2 is already in. 2024-09-15T20:28:08.682 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T20:28:08.682 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2024-09-15T20:28:08.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:28:08.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:28:08.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:28:08.683 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T20:28:08.683 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:28:08.683 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:28:08.684 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:28:08.684 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:28:08.684 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:28:08.749 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T20:28:08.749 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:28:08.750 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:28:08.750 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:28:08.750 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:28:08.750 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:28:09.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:28:09.241 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:28:09.241 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:28:09.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:28:09.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:28:09.241 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:28:09.405 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822660 2024-09-15T20:28:09.405 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822660 2024-09-15T20:28:09.405 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660' 2024-09-15T20:28:09.405 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:28:09.405 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:28:09.526 INFO:tasks.workunit.client.0.smithi139.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-15T20:28:09.528 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-15T20:28:09.528 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-15T20:28:09.528 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-15T20:28:09.529 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:28:09.529 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:28:09.697 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757251 2024-09-15T20:28:09.710 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757251 2024-09-15T20:28:09.710 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-09-15T20:28:09.710 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:28:09.710 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-154618822660 2024-09-15T20:28:09.710 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:28:09.710 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:28:09.711 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-154618822660 2024-09-15T20:28:09.711 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:28:09.711 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822660 2024-09-15T20:28:09.711 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 154618822660 2024-09-15T20:28:09.712 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-09-15T20:28:09.712 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:28:10.154 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822658 -lt 154618822660 2024-09-15T20:28:10.155 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:28:11.155 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:28:11.156 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:28:11.632 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822660 -lt 154618822660 2024-09-15T20:28:11.632 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:28:11.633 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-163208757251 2024-09-15T20:28:11.633 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:28:11.635 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:28:11.635 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-163208757251 2024-09-15T20:28:11.635 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:28:11.637 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757251 2024-09-15T20:28:11.637 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-09-15T20:28:11.637 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 163208757251 2024-09-15T20:28:11.637 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:28:12.077 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757251 -lt 163208757251 2024-09-15T20:28:12.077 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:28:12.078 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:28:12.078 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:28:12.668 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:28:12.669 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:28:12.669 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:28:12.669 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:28:12.669 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:28:12.670 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:28:12.670 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:28:12.670 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:28:13.080 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:28:13.081 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:28:13.081 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:28:13.081 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:28:13.644 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:28:13.644 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T20:28:13.644 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:28:13.644 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T20:28:13.644 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:28:13.644 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:28:13.745 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:28:13.745 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:28:13.746 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:28:13.746 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:28:13.746 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:28:13.747 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:28:13.747 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:28:13.747 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:28:14.178 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:28:14.179 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:28:14.179 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:28:14.179 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:28:14.752 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:28:14.752 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:28:14.752 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:28:14.752 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:28:14.752 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:28:14.753 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:28:14.753 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:28:14.753 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:28:14.753 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:28:15.298 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:28:15.299 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:28:15.299 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-15T20:28:15.299 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:28:15.299 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:28:15.500 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:28:15.500 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:28:15.500 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:28:15.500 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:28:15.501 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:28:15.501 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:28:15.501 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:28:15.501 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:28:15.913 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:28:15.914 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:28:15.914 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:28:15.914 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:28:16.457 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:28:16.457 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:28:16.457 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:28:16.457 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:28:16.457 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:28:16.457 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:28:16.457 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:28:16.458 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:28:16.458 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:28:17.066 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:28:17.066 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:28:17.066 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-15T20:28:17.066 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:28:17.066 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T20:28:17.467 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:28:17.467 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:28:17.468 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:28:17.468 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:28:17.468 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:28:17.468 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:28:17.469 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:28:17.469 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:28:17.901 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:28:17.902 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:28:17.902 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:28:17.902 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:28:18.484 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:28:18.484 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:28:18.484 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:28:18.484 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:28:18.484 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:28:18.484 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:28:18.484 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:28:18.485 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:28:18.485 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:28:19.058 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:28:19.059 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:28:19.059 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-15T20:28:19.059 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:28:19.059 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-15T20:28:19.860 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:28:19.860 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:28:19.860 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:28:19.861 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:28:19.861 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:28:19.861 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:28:19.861 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:28:19.861 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:28:20.300 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:28:20.301 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:28:20.301 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:28:20.301 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:28:20.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:28:20.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:28:20.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:28:20.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:28:20.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:28:20.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:28:20.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:28:20.857 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:28:20.857 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:28:21.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:28:21.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:28:21.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-15T20:28:21.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:28:21.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-15T20:28:23.029 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:28:23.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:28:23.030 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:28:23.030 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:28:23.031 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:28:23.031 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:28:23.031 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:28:23.031 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:28:23.456 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:28:23.457 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:28:23.457 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:28:23.457 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:28:24.039 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:28:24.039 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:28:24.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:28:24.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:28:24.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:28:24.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:28:24.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:28:24.040 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:28:24.041 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:28:24.612 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:28:24.613 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:28:24.613 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-15T20:28:24.613 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:28:24.613 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-15T20:28:27.814 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:28:27.814 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:28:27.815 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:28:27.815 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:28:27.815 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:28:27.816 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:28:27.816 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:28:27.816 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:28:28.236 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:28:28.236 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:28:28.237 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:28:28.237 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:28:28.835 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:28:28.835 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:28:28.835 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:28:28.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:28:28.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:28:28.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:28:28.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:28:28.836 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:28:28.836 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:28:29.381 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:28:29.382 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:28:29.382 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-15T20:28:29.382 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:28:29.382 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-15T20:28:35.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:28:35.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:28:35.784 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:28:35.784 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:28:35.784 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:28:35.784 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:28:35.784 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:28:35.785 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:28:36.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:28:36.225 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:28:36.225 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:28:36.225 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:28:36.811 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:28:36.811 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:28:36.811 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:28:36.811 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:28:36.811 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:28:36.811 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:28:36.811 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:28:36.811 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:28:36.811 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:28:37.374 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:28:37.374 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:28:37.374 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-15T20:28:37.374 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:28:37.374 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-15T20:28:44.675 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:28:44.675 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:28:44.676 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:28:44.676 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:28:44.676 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:28:44.676 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:28:44.677 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:28:44.677 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:28:45.118 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:28:45.119 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:28:45.119 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:28:45.119 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:28:45.666 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:28:45.667 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:28:45.667 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:28:45.667 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:28:45.667 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:28:45.667 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:28:45.667 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:28:45.668 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:28:45.668 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:28:46.246 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:28:46.246 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:28:46.246 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-15T20:28:46.246 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-15T20:28:46.890 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:28:46.891 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_fingerprint": "2482eff2-b654-43ea-b114-58257cb4aa0c", 2024-09-15T20:28:46.891 INFO:tasks.workunit.client.0.smithi139.stdout: "version": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:28:46.891 INFO:tasks.workunit.client.0.smithi139.stdout: "commit": "fc3e0bff7468fd55478adbae7f75508b749124b6", 2024-09-15T20:28:46.891 INFO:tasks.workunit.client.0.smithi139.stdout: "timestamp": "2024-09-15T20:28:46.888436+0000", 2024-09-15T20:28:46.891 INFO:tasks.workunit.client.0.smithi139.stdout: "tag": "", 2024-09-15T20:28:46.891 INFO:tasks.workunit.client.0.smithi139.stdout: "health": { 2024-09-15T20:28:46.891 INFO:tasks.workunit.client.0.smithi139.stdout: "status": "HEALTH_WARN", 2024-09-15T20:28:46.891 INFO:tasks.workunit.client.0.smithi139.stdout: "checks": { 2024-09-15T20:28:46.891 INFO:tasks.workunit.client.0.smithi139.stdout: "OSDMAP_FLAGS": { 2024-09-15T20:28:46.891 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:28:46.891 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:28:46.891 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-15T20:28:46.891 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 17 2024-09-15T20:28:46.891 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.891 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [], 2024-09-15T20:28:46.892 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:28:46.892 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.892 INFO:tasks.workunit.client.0.smithi139.stdout: "OSD_DOWN": { 2024-09-15T20:28:46.892 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:28:46.892 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:28:46.892 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "1 osds down", 2024-09-15T20:28:46.892 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 1 2024-09-15T20:28:46.892 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.892 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:28:46.892 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.892 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "osd.1 (root=default,host=smithi139) is down" 2024-09-15T20:28:46.892 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.892 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:28:46.892 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:28:46.892 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.893 INFO:tasks.workunit.client.0.smithi139.stdout: "PG_DEGRADED": { 2024-09-15T20:28:46.893 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:28:46.893 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:28:46.893 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-09-15T20:28:46.893 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 1 2024-09-15T20:28:46.893 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.893 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:28:46.893 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.893 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-15T20:28:46.893 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.893 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:28:46.893 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:28:46.893 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.894 INFO:tasks.workunit.client.0.smithi139.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-15T20:28:46.894 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:28:46.894 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:28:46.894 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-15T20:28:46.894 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 1 2024-09-15T20:28:46.894 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.894 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:28:46.894 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.894 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "application not enabled on pool 'test'" 2024-09-15T20:28:46.894 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.894 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.894 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-15T20:28:46.894 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.895 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:28:46.895 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:28:46.895 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.895 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.895 INFO:tasks.workunit.client.0.smithi139.stdout: "mutes": [] 2024-09-15T20:28:46.895 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.895 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap_first_committed": 1, 2024-09-15T20:28:46.895 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap_last_committed": 1, 2024-09-15T20:28:46.895 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap": { 2024-09-15T20:28:46.895 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 1, 2024-09-15T20:28:46.895 INFO:tasks.workunit.client.0.smithi139.stdout: "fsid": "0e220df1-ac9c-433e-8885-cf9f7aab293e", 2024-09-15T20:28:46.895 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:25:17.608977Z", 2024-09-15T20:28:46.895 INFO:tasks.workunit.client.0.smithi139.stdout: "created": "2024-09-15T20:25:17.608977Z", 2024-09-15T20:28:46.895 INFO:tasks.workunit.client.0.smithi139.stdout: "min_mon_release": 19, 2024-09-15T20:28:46.896 INFO:tasks.workunit.client.0.smithi139.stdout: "min_mon_release_name": "squid", 2024-09-15T20:28:46.896 INFO:tasks.workunit.client.0.smithi139.stdout: "election_strategy": 1, 2024-09-15T20:28:46.896 INFO:tasks.workunit.client.0.smithi139.stdout: "disallowed_leaders: ": "", 2024-09-15T20:28:46.896 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode": false, 2024-09-15T20:28:46.896 INFO:tasks.workunit.client.0.smithi139.stdout: "tiebreaker_mon": "", 2024-09-15T20:28:46.896 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_ranks: ": "", 2024-09-15T20:28:46.896 INFO:tasks.workunit.client.0.smithi139.stdout: "features": { 2024-09-15T20:28:46.896 INFO:tasks.workunit.client.0.smithi139.stdout: "persistent": [ 2024-09-15T20:28:46.896 INFO:tasks.workunit.client.0.smithi139.stdout: "kraken", 2024-09-15T20:28:46.896 INFO:tasks.workunit.client.0.smithi139.stdout: "luminous", 2024-09-15T20:28:46.896 INFO:tasks.workunit.client.0.smithi139.stdout: "mimic", 2024-09-15T20:28:46.896 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap-prune", 2024-09-15T20:28:46.896 INFO:tasks.workunit.client.0.smithi139.stdout: "nautilus", 2024-09-15T20:28:46.897 INFO:tasks.workunit.client.0.smithi139.stdout: "octopus", 2024-09-15T20:28:46.897 INFO:tasks.workunit.client.0.smithi139.stdout: "pacific", 2024-09-15T20:28:46.897 INFO:tasks.workunit.client.0.smithi139.stdout: "elector-pinging", 2024-09-15T20:28:46.897 INFO:tasks.workunit.client.0.smithi139.stdout: "quincy", 2024-09-15T20:28:46.897 INFO:tasks.workunit.client.0.smithi139.stdout: "reef", 2024-09-15T20:28:46.897 INFO:tasks.workunit.client.0.smithi139.stdout: "squid" 2024-09-15T20:28:46.897 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:28:46.897 INFO:tasks.workunit.client.0.smithi139.stdout: "optional": [] 2024-09-15T20:28:46.897 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.897 INFO:tasks.workunit.client.0.smithi139.stdout: "mons": [ 2024-09-15T20:28:46.897 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.897 INFO:tasks.workunit.client.0.smithi139.stdout: "rank": 0, 2024-09-15T20:28:46.897 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "a", 2024-09-15T20:28:46.898 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:28:46.898 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:28:46.898 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.898 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:28:46.898 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:7115", 2024-09-15T20:28:46.898 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 0 2024-09-15T20:28:46.898 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.898 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.898 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.898 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:7115/0", 2024-09-15T20:28:46.898 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-15T20:28:46.898 INFO:tasks.workunit.client.0.smithi139.stdout: "priority": 0, 2024-09-15T20:28:46.898 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 0, 2024-09-15T20:28:46.899 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_location": "{}" 2024-09-15T20:28:46.899 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.899 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.899 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.899 INFO:tasks.workunit.client.0.smithi139.stdout: "quorum": [ 2024-09-15T20:28:46.899 INFO:tasks.workunit.client.0.smithi139.stdout: 0 2024-09-15T20:28:46.899 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:28:46.899 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap": { 2024-09-15T20:28:46.899 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 41, 2024-09-15T20:28:46.899 INFO:tasks.workunit.client.0.smithi139.stdout: "fsid": "0e220df1-ac9c-433e-8885-cf9f7aab293e", 2024-09-15T20:28:46.899 INFO:tasks.workunit.client.0.smithi139.stdout: "created": "2024-09-15T20:25:18.567361+0000", 2024-09-15T20:28:46.899 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:28:08.791188+0000", 2024-09-15T20:28:46.899 INFO:tasks.workunit.client.0.smithi139.stdout: "last_up_change": "2024-09-15T20:28:06.539868+0000", 2024-09-15T20:28:46.899 INFO:tasks.workunit.client.0.smithi139.stdout: "last_in_change": "2024-09-15T20:28:08.437093+0000", 2024-09-15T20:28:46.899 INFO:tasks.workunit.client.0.smithi139.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T20:28:46.899 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_num": 5800384, 2024-09-15T20:28:46.900 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_set": [ 2024-09-15T20:28:46.900 INFO:tasks.workunit.client.0.smithi139.stdout: "nodown", 2024-09-15T20:28:46.900 INFO:tasks.workunit.client.0.smithi139.stdout: "noin", 2024-09-15T20:28:46.900 INFO:tasks.workunit.client.0.smithi139.stdout: "noout", 2024-09-15T20:28:46.900 INFO:tasks.workunit.client.0.smithi139.stdout: "pglog_hardlimit", 2024-09-15T20:28:46.900 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snapdirs", 2024-09-15T20:28:46.900 INFO:tasks.workunit.client.0.smithi139.stdout: "recovery_deletes", 2024-09-15T20:28:46.900 INFO:tasks.workunit.client.0.smithi139.stdout: "sortbitwise" 2024-09-15T20:28:46.900 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:28:46.900 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_version": 9, 2024-09-15T20:28:46.900 INFO:tasks.workunit.client.0.smithi139.stdout: "full_ratio": 0.99000000953674316, 2024-09-15T20:28:46.900 INFO:tasks.workunit.client.0.smithi139.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-15T20:28:46.900 INFO:tasks.workunit.client.0.smithi139.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T20:28:46.900 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_snapshot": "", 2024-09-15T20:28:46.901 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_max": 1, 2024-09-15T20:28:46.901 INFO:tasks.workunit.client.0.smithi139.stdout: "max_osd": 3, 2024-09-15T20:28:46.901 INFO:tasks.workunit.client.0.smithi139.stdout: "require_min_compat_client": "luminous", 2024-09-15T20:28:46.901 INFO:tasks.workunit.client.0.smithi139.stdout: "min_compat_client": "jewel", 2024-09-15T20:28:46.901 INFO:tasks.workunit.client.0.smithi139.stdout: "require_osd_release": "squid", 2024-09-15T20:28:46.901 INFO:tasks.workunit.client.0.smithi139.stdout: "allow_crimson": false, 2024-09-15T20:28:46.901 INFO:tasks.workunit.client.0.smithi139.stdout: "pools": [ 2024-09-15T20:28:46.901 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.901 INFO:tasks.workunit.client.0.smithi139.stdout: "pool": 1, 2024-09-15T20:28:46.901 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_name": "test", 2024-09-15T20:28:46.901 INFO:tasks.workunit.client.0.smithi139.stdout: "create_time": "2024-09-15T20:27:08.011839+0000", 2024-09-15T20:28:46.901 INFO:tasks.workunit.client.0.smithi139.stdout: "flags": 1, 2024-09-15T20:28:46.901 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_names": "hashpspool", 2024-09-15T20:28:46.901 INFO:tasks.workunit.client.0.smithi139.stdout: "type": 1, 2024-09-15T20:28:46.901 INFO:tasks.workunit.client.0.smithi139.stdout: "size": 3, 2024-09-15T20:28:46.902 INFO:tasks.workunit.client.0.smithi139.stdout: "min_size": 2, 2024-09-15T20:28:46.902 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_rule": 0, 2024-09-15T20:28:46.902 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_count": 0, 2024-09-15T20:28:46.902 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_target": 0, 2024-09-15T20:28:46.902 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T20:28:46.902 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T20:28:46.902 INFO:tasks.workunit.client.0.smithi139.stdout: "is_stretch_pool": false, 2024-09-15T20:28:46.902 INFO:tasks.workunit.client.0.smithi139.stdout: "object_hash": 2, 2024-09-15T20:28:46.902 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_autoscale_mode": "off", 2024-09-15T20:28:46.902 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num": 1, 2024-09-15T20:28:46.902 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_placement_num": 1, 2024-09-15T20:28:46.902 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_placement_num_target": 1, 2024-09-15T20:28:46.902 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num_target": 1, 2024-09-15T20:28:46.902 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num_pending": 1, 2024-09-15T20:28:46.903 INFO:tasks.workunit.client.0.smithi139.stdout: "last_pg_merge_meta": { 2024-09-15T20:28:46.903 INFO:tasks.workunit.client.0.smithi139.stdout: "source_pgid": "0.0", 2024-09-15T20:28:46.903 INFO:tasks.workunit.client.0.smithi139.stdout: "ready_epoch": 0, 2024-09-15T20:28:46.903 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_started": 0, 2024-09-15T20:28:46.903 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_clean": 0, 2024-09-15T20:28:46.903 INFO:tasks.workunit.client.0.smithi139.stdout: "source_version": "0'0", 2024-09-15T20:28:46.903 INFO:tasks.workunit.client.0.smithi139.stdout: "target_version": "0'0" 2024-09-15T20:28:46.903 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.903 INFO:tasks.workunit.client.0.smithi139.stdout: "last_change": "29", 2024-09-15T20:28:46.903 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend": "0", 2024-09-15T20:28:46.903 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T20:28:46.903 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T20:28:46.903 INFO:tasks.workunit.client.0.smithi139.stdout: "auid": 0, 2024-09-15T20:28:46.904 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_mode": "selfmanaged", 2024-09-15T20:28:46.904 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_seq": 0, 2024-09-15T20:28:46.904 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_epoch": 0, 2024-09-15T20:28:46.904 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_snaps": [], 2024-09-15T20:28:46.904 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_snaps": "[]", 2024-09-15T20:28:46.904 INFO:tasks.workunit.client.0.smithi139.stdout: "quota_max_bytes": 0, 2024-09-15T20:28:46.904 INFO:tasks.workunit.client.0.smithi139.stdout: "quota_max_objects": 0, 2024-09-15T20:28:46.904 INFO:tasks.workunit.client.0.smithi139.stdout: "tiers": [], 2024-09-15T20:28:46.904 INFO:tasks.workunit.client.0.smithi139.stdout: "tier_of": -1, 2024-09-15T20:28:46.904 INFO:tasks.workunit.client.0.smithi139.stdout: "read_tier": -1, 2024-09-15T20:28:46.904 INFO:tasks.workunit.client.0.smithi139.stdout: "write_tier": -1, 2024-09-15T20:28:46.904 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_mode": "none", 2024-09-15T20:28:46.904 INFO:tasks.workunit.client.0.smithi139.stdout: "target_max_bytes": 0, 2024-09-15T20:28:46.905 INFO:tasks.workunit.client.0.smithi139.stdout: "target_max_objects": 0, 2024-09-15T20:28:46.905 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T20:28:46.905 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T20:28:46.905 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T20:28:46.905 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_min_flush_age": 0, 2024-09-15T20:28:46.905 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_min_evict_age": 0, 2024-09-15T20:28:46.905 INFO:tasks.workunit.client.0.smithi139.stdout: "erasure_code_profile": "", 2024-09-15T20:28:46.905 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_params": { 2024-09-15T20:28:46.905 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "none" 2024-09-15T20:28:46.905 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.905 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_period": 0, 2024-09-15T20:28:46.905 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_count": 0, 2024-09-15T20:28:46.905 INFO:tasks.workunit.client.0.smithi139.stdout: "use_gmt_hitset": true, 2024-09-15T20:28:46.905 INFO:tasks.workunit.client.0.smithi139.stdout: "min_read_recency_for_promote": 0, 2024-09-15T20:28:46.906 INFO:tasks.workunit.client.0.smithi139.stdout: "min_write_recency_for_promote": 0, 2024-09-15T20:28:46.906 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T20:28:46.906 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_search_last_n": 0, 2024-09-15T20:28:46.906 INFO:tasks.workunit.client.0.smithi139.stdout: "grade_table": [], 2024-09-15T20:28:46.906 INFO:tasks.workunit.client.0.smithi139.stdout: "stripe_width": 0, 2024-09-15T20:28:46.906 INFO:tasks.workunit.client.0.smithi139.stdout: "expected_num_objects": 0, 2024-09-15T20:28:46.906 INFO:tasks.workunit.client.0.smithi139.stdout: "fast_read": false, 2024-09-15T20:28:46.906 INFO:tasks.workunit.client.0.smithi139.stdout: "options": {}, 2024-09-15T20:28:46.906 INFO:tasks.workunit.client.0.smithi139.stdout: "application_metadata": {}, 2024-09-15T20:28:46.906 INFO:tasks.workunit.client.0.smithi139.stdout: "read_balance": { 2024-09-15T20:28:46.906 INFO:tasks.workunit.client.0.smithi139.stdout: "score_type": "Fair distribution", 2024-09-15T20:28:46.906 INFO:tasks.workunit.client.0.smithi139.stdout: "score_acting": 2.9900000095367432, 2024-09-15T20:28:46.906 INFO:tasks.workunit.client.0.smithi139.stdout: "score_stable": 2.9900000095367432, 2024-09-15T20:28:46.906 INFO:tasks.workunit.client.0.smithi139.stdout: "optimal_score": 0.67000001668930054, 2024-09-15T20:28:46.907 INFO:tasks.workunit.client.0.smithi139.stdout: "raw_score_acting": 2, 2024-09-15T20:28:46.907 INFO:tasks.workunit.client.0.smithi139.stdout: "raw_score_stable": 2, 2024-09-15T20:28:46.907 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity_weighted": 1, 2024-09-15T20:28:46.907 INFO:tasks.workunit.client.0.smithi139.stdout: "average_primary_affinity": 1, 2024-09-15T20:28:46.907 INFO:tasks.workunit.client.0.smithi139.stdout: "average_primary_affinity_weighted": 1 2024-09-15T20:28:46.907 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.907 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.907 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:28:46.907 INFO:tasks.workunit.client.0.smithi139.stdout: "osds": [ 2024-09-15T20:28:46.907 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.907 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:28:46.907 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "cdf05e2d-6e34-45ea-89d8-b5b29cbb7f44", 2024-09-15T20:28:46.907 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 1, 2024-09-15T20:28:46.907 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:28:46.908 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:28:46.908 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:28:46.908 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 5, 2024-09-15T20:28:46.908 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 29, 2024-09-15T20:28:46.908 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 36, 2024-09-15T20:28:46.908 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 40, 2024-09-15T20:28:46.908 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 30, 2024-09-15T20:28:46.908 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:28:46.908 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:28:46.908 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:28:46.908 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.908 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:28:46.908 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6802", 2024-09-15T20:28:46.908 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3285624630 2024-09-15T20:28:46.909 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.909 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.909 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:28:46.909 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6803", 2024-09-15T20:28:46.909 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3285624630 2024-09-15T20:28:46.909 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.909 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.909 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.909 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:28:46.909 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:28:46.909 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.909 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:28:46.909 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6804", 2024-09-15T20:28:46.909 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3285624630 2024-09-15T20:28:46.910 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.910 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.910 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:28:46.910 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6805", 2024-09-15T20:28:46.910 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3285624630 2024-09-15T20:28:46.910 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.910 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.910 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.910 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:28:46.910 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:28:46.910 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.910 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:28:46.910 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6808", 2024-09-15T20:28:46.910 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3285624630 2024-09-15T20:28:46.910 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.911 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.911 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:28:46.911 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6809", 2024-09-15T20:28:46.911 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3285624630 2024-09-15T20:28:46.911 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.911 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.911 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.911 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:28:46.911 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:28:46.911 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.911 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:28:46.911 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6806", 2024-09-15T20:28:46.912 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3285624630 2024-09-15T20:28:46.912 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.912 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.912 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:28:46.912 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6807", 2024-09-15T20:28:46.912 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3285624630 2024-09-15T20:28:46.912 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.912 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.912 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.912 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6803/3285624630", 2024-09-15T20:28:46.912 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6805/3285624630", 2024-09-15T20:28:46.912 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3285624630", 2024-09-15T20:28:46.912 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3285624630", 2024-09-15T20:28:46.912 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:28:46.913 INFO:tasks.workunit.client.0.smithi139.stdout: "exists", 2024-09-15T20:28:46.913 INFO:tasks.workunit.client.0.smithi139.stdout: "up" 2024-09-15T20:28:46.913 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.913 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.913 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.913 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:28:46.913 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "95c45dfa-5b44-4aaf-9fc1-ebb69bbe1237", 2024-09-15T20:28:46.913 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 0, 2024-09-15T20:28:46.913 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:28:46.913 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:28:46.913 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:28:46.913 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 0, 2024-09-15T20:28:46.913 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 0, 2024-09-15T20:28:46.913 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 10, 2024-09-15T20:28:46.914 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 23, 2024-09-15T20:28:46.914 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 31, 2024-09-15T20:28:46.914 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:28:46.914 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:28:46.914 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:28:46.914 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.914 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:28:46.914 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6810", 2024-09-15T20:28:46.914 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3778108260 2024-09-15T20:28:46.914 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.914 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.914 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:28:46.914 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6811", 2024-09-15T20:28:46.914 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3778108260 2024-09-15T20:28:46.915 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.915 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.915 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.915 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:28:46.915 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:28:46.915 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.915 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:28:46.915 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6812", 2024-09-15T20:28:46.915 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3778108260 2024-09-15T20:28:46.915 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.915 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.915 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:28:46.915 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6813", 2024-09-15T20:28:46.916 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3778108260 2024-09-15T20:28:46.916 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.916 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.916 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.916 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:28:46.916 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:28:46.916 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.916 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:28:46.916 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6816", 2024-09-15T20:28:46.916 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3778108260 2024-09-15T20:28:46.916 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.916 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.916 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:28:46.916 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6817", 2024-09-15T20:28:46.916 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3778108260 2024-09-15T20:28:46.917 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.917 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.917 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.917 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:28:46.917 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:28:46.917 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.917 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:28:46.917 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6814", 2024-09-15T20:28:46.917 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3778108260 2024-09-15T20:28:46.917 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.917 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.917 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:28:46.917 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6815", 2024-09-15T20:28:46.917 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3778108260 2024-09-15T20:28:46.918 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.918 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.918 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.918 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6811/3778108260", 2024-09-15T20:28:46.918 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6813/3778108260", 2024-09-15T20:28:46.918 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3778108260", 2024-09-15T20:28:46.918 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3778108260", 2024-09-15T20:28:46.918 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:28:46.918 INFO:tasks.workunit.client.0.smithi139.stdout: "exists" 2024-09-15T20:28:46.918 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.918 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.918 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.918 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:28:46.918 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "9d0447d3-39ad-48fb-8e36-2d8f2ba5c93b", 2024-09-15T20:28:46.918 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 1, 2024-09-15T20:28:46.919 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:28:46.919 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:28:46.919 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:28:46.919 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 15, 2024-09-15T20:28:46.919 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 29, 2024-09-15T20:28:46.919 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 38, 2024-09-15T20:28:46.919 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 0, 2024-09-15T20:28:46.919 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 32, 2024-09-15T20:28:46.919 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:28:46.919 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:28:46.919 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:28:46.919 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.919 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:28:46.920 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6810", 2024-09-15T20:28:46.920 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 919482650 2024-09-15T20:28:46.920 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.920 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.920 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:28:46.920 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6811", 2024-09-15T20:28:46.920 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 919482650 2024-09-15T20:28:46.920 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.920 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.920 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.920 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:28:46.920 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:28:46.920 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.921 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:28:46.921 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6812", 2024-09-15T20:28:46.921 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 919482650 2024-09-15T20:28:46.921 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.921 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.921 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:28:46.921 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6813", 2024-09-15T20:28:46.921 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 919482650 2024-09-15T20:28:46.921 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.921 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.921 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.921 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:28:46.921 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:28:46.921 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.921 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:28:46.924 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6816", 2024-09-15T20:28:46.924 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 919482650 2024-09-15T20:28:46.924 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.924 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.924 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:28:46.924 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6817", 2024-09-15T20:28:46.924 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 919482650 2024-09-15T20:28:46.924 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.924 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.925 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.925 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:28:46.925 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:28:46.925 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.925 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:28:46.925 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6814", 2024-09-15T20:28:46.925 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 919482650 2024-09-15T20:28:46.925 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.925 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.925 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:28:46.925 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6815", 2024-09-15T20:28:46.925 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 919482650 2024-09-15T20:28:46.925 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.925 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.926 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.926 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6811/919482650", 2024-09-15T20:28:46.926 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6813/919482650", 2024-09-15T20:28:46.926 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6817/919482650", 2024-09-15T20:28:46.926 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6815/919482650", 2024-09-15T20:28:46.926 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:28:46.926 INFO:tasks.workunit.client.0.smithi139.stdout: "exists", 2024-09-15T20:28:46.926 INFO:tasks.workunit.client.0.smithi139.stdout: "up" 2024-09-15T20:28:46.926 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.926 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.926 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:28:46.926 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_xinfo": [ 2024-09-15T20:28:46.927 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.927 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:28:46.927 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:27:53.768600+0000", 2024-09-15T20:28:46.927 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:28:46.927 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:28:46.927 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:28:46.927 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:28:46.927 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:25:51.861123+0000", 2024-09-15T20:28:46.927 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:28:46.927 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.927 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.927 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:28:46.927 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:27:54.353752+0000", 2024-09-15T20:28:46.927 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:28:46.928 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:28:46.928 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:28:46.928 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:28:46.928 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:26:29.145987+0000", 2024-09-15T20:28:46.928 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:28:46.928 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.928 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.928 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:28:46.928 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:27:55.012420+0000", 2024-09-15T20:28:46.928 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:28:46.928 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:28:46.928 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:28:46.928 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:28:46.929 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:27:03.968270+0000", 2024-09-15T20:28:46.929 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:28:46.929 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.929 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:28:46.929 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap": [], 2024-09-15T20:28:46.929 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap_items": [], 2024-09-15T20:28:46.929 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap_primaries": [], 2024-09-15T20:28:46.929 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_temp": [], 2024-09-15T20:28:46.929 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_temp": [], 2024-09-15T20:28:46.929 INFO:tasks.workunit.client.0.smithi139.stdout: "blocklist": {}, 2024-09-15T20:28:46.929 INFO:tasks.workunit.client.0.smithi139.stdout: "range_blocklist": {}, 2024-09-15T20:28:46.929 INFO:tasks.workunit.client.0.smithi139.stdout: "erasure_code_profiles": { 2024-09-15T20:28:46.929 INFO:tasks.workunit.client.0.smithi139.stdout: "default": { 2024-09-15T20:28:46.929 INFO:tasks.workunit.client.0.smithi139.stdout: "k": "2", 2024-09-15T20:28:46.929 INFO:tasks.workunit.client.0.smithi139.stdout: "m": "2", 2024-09-15T20:28:46.930 INFO:tasks.workunit.client.0.smithi139.stdout: "plugin": "jerasure", 2024-09-15T20:28:46.930 INFO:tasks.workunit.client.0.smithi139.stdout: "technique": "reed_sol_van" 2024-09-15T20:28:46.930 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.930 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.930 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_snaps_queue": [], 2024-09-15T20:28:46.930 INFO:tasks.workunit.client.0.smithi139.stdout: "new_removed_snaps": [], 2024-09-15T20:28:46.930 INFO:tasks.workunit.client.0.smithi139.stdout: "new_purged_snaps": [], 2024-09-15T20:28:46.930 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_node_flags": {}, 2024-09-15T20:28:46.930 INFO:tasks.workunit.client.0.smithi139.stdout: "device_class_flags": {}, 2024-09-15T20:28:46.930 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode": { 2024-09-15T20:28:46.930 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode_enabled": false, 2024-09-15T20:28:46.930 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_bucket_count": 0, 2024-09-15T20:28:46.930 INFO:tasks.workunit.client.0.smithi139.stdout: "degraded_stretch_mode": 0, 2024-09-15T20:28:46.930 INFO:tasks.workunit.client.0.smithi139.stdout: "recovering_stretch_mode": 0, 2024-09-15T20:28:46.930 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode_bucket": 0 2024-09-15T20:28:46.931 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.931 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.931 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_metadata": [ 2024-09-15T20:28:46.931 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.931 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:28:46.931 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:28:46.931 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6804/3285624630,v1:127.0.0.1:6805/3285624630]", 2024-09-15T20:28:46.931 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:28:46.931 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:28:46.931 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:28:46.931 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:28:46.931 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:28:46.931 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:28:46.931 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:28:46.931 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:28:46.932 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:28:46.932 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:28:46.932 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:28:46.932 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-15T20:28:46.932 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:28:46.932 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:28:46.932 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:28:46.932 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:28:46.932 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:28:46.932 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:28:46.932 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:28:46.932 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:28:46.932 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:28:46.932 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:28:46.932 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:25:25.232714Z", 2024-09-15T20:28:46.932 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:28:46.933 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:28:46.933 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:28:46.933 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:28:46.933 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:28:46.933 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:28:46.933 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:28:46.933 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6802/3285624630,v1:127.0.0.1:6803/3285624630]", 2024-09-15T20:28:46.933 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:28:46.933 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3285624630,v1:127.0.0.1:6809/3285624630]", 2024-09-15T20:28:46.933 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3285624630,v1:127.0.0.1:6807/3285624630]", 2024-09-15T20:28:46.933 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:28:46.933 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:28:46.933 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:28:46.933 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:28:46.933 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:28:46.934 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:28:46.934 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:28:46.934 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:28:46.934 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:28:46.934 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/0", 2024-09-15T20:28:46.934 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:28:46.934 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:28:46.934 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:28:46.934 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.934 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.934 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:28:46.934 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:28:46.934 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6812/3778108260,v1:127.0.0.1:6813/3778108260]", 2024-09-15T20:28:46.934 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:28:46.935 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:28:46.935 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:28:46.935 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:28:46.935 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:28:46.935 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:28:46.935 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:28:46.935 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:28:46.935 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:28:46.935 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:28:46.935 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:28:46.935 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-15T20:28:46.935 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:28:46.935 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:28:46.935 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:28:46.935 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:28:46.935 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:28:46.936 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:28:46.936 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:28:46.936 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:28:46.936 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:28:46.936 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:28:46.936 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:25:59.052762Z", 2024-09-15T20:28:46.936 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:28:46.936 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:28:46.936 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:28:46.936 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:28:46.936 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:28:46.936 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:28:46.936 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:28:46.936 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6810/3778108260,v1:127.0.0.1:6811/3778108260]", 2024-09-15T20:28:46.936 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:28:46.937 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3778108260,v1:127.0.0.1:6817/3778108260]", 2024-09-15T20:28:46.937 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3778108260,v1:127.0.0.1:6815/3778108260]", 2024-09-15T20:28:46.937 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:28:46.937 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:28:46.937 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:28:46.937 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:28:46.937 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:28:46.937 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:28:46.937 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:28:46.937 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:28:46.937 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:28:46.937 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/1", 2024-09-15T20:28:46.937 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:28:46.938 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:28:46.938 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:28:46.938 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.938 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.938 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:28:46.938 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:28:46.938 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6812/919482650,v1:127.0.0.1:6813/919482650]", 2024-09-15T20:28:46.938 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:28:46.938 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:28:46.938 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:28:46.938 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:28:46.938 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:28:46.938 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:28:46.938 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:28:46.939 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:28:46.939 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:28:46.939 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:28:46.939 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:28:46.939 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-15T20:28:46.939 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:28:46.939 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:28:46.939 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:28:46.939 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:28:46.939 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:28:46.939 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:28:46.939 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:28:46.939 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:28:46.940 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:28:46.940 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:28:46.940 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:26:35.637514Z", 2024-09-15T20:28:46.940 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:28:46.940 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:28:46.940 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:28:46.940 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:28:46.940 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:28:46.940 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:28:46.940 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:28:46.940 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6810/919482650,v1:127.0.0.1:6811/919482650]", 2024-09-15T20:28:46.940 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:28:46.940 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/919482650,v1:127.0.0.1:6817/919482650]", 2024-09-15T20:28:46.940 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/919482650,v1:127.0.0.1:6815/919482650]", 2024-09-15T20:28:46.941 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:28:46.941 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:28:46.941 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:28:46.941 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:28:46.941 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:28:46.941 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:28:46.941 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:28:46.941 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:28:46.941 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:28:46.941 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/2", 2024-09-15T20:28:46.941 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:28:46.941 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:28:46.941 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:28:46.942 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.942 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:28:46.942 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_clean_epochs": { 2024-09-15T20:28:46.942 INFO:tasks.workunit.client.0.smithi139.stdout: "min_last_epoch_clean": 0, 2024-09-15T20:28:46.942 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_clean": { 2024-09-15T20:28:46.942 INFO:tasks.workunit.client.0.smithi139.stdout: "per_pool": [] 2024-09-15T20:28:46.942 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.942 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_epochs": [ 2024-09-15T20:28:46.942 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.942 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:28:46.942 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 36 2024-09-15T20:28:46.942 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.942 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.943 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:28:46.943 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 38 2024-09-15T20:28:46.943 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.943 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.943 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.943 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_first_committed": 1, 2024-09-15T20:28:46.943 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_last_committed": 41, 2024-09-15T20:28:46.943 INFO:tasks.workunit.client.0.smithi139.stdout: "crushmap": { 2024-09-15T20:28:46.943 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": [ 2024-09-15T20:28:46.943 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.943 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:28:46.943 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.0", 2024-09-15T20:28:46.944 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:28:46.944 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.944 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.944 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:28:46.944 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.1", 2024-09-15T20:28:46.944 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:28:46.944 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.944 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.944 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:28:46.944 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.2", 2024-09-15T20:28:46.944 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:28:46.944 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.944 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:28:46.944 INFO:tasks.workunit.client.0.smithi139.stdout: "types": [ 2024-09-15T20:28:46.945 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.945 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 0, 2024-09-15T20:28:46.945 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd" 2024-09-15T20:28:46.945 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.945 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.945 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:28:46.945 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "host" 2024-09-15T20:28:46.945 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.945 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.945 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 2, 2024-09-15T20:28:46.945 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "chassis" 2024-09-15T20:28:46.945 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.945 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.946 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 3, 2024-09-15T20:28:46.946 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "rack" 2024-09-15T20:28:46.946 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.946 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.946 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 4, 2024-09-15T20:28:46.946 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "row" 2024-09-15T20:28:46.946 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.946 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.946 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 5, 2024-09-15T20:28:46.946 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "pdu" 2024-09-15T20:28:46.946 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.946 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.946 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 6, 2024-09-15T20:28:46.947 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "pod" 2024-09-15T20:28:46.947 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.947 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.947 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 7, 2024-09-15T20:28:46.947 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "room" 2024-09-15T20:28:46.947 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.947 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.947 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 8, 2024-09-15T20:28:46.947 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "datacenter" 2024-09-15T20:28:46.947 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.947 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.947 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 9, 2024-09-15T20:28:46.947 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "zone" 2024-09-15T20:28:46.947 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.948 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.948 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 10, 2024-09-15T20:28:46.948 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "region" 2024-09-15T20:28:46.948 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.948 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.948 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:28:46.948 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "root" 2024-09-15T20:28:46.948 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.948 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:28:46.948 INFO:tasks.workunit.client.0.smithi139.stdout: "buckets": [ 2024-09-15T20:28:46.948 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.948 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -1, 2024-09-15T20:28:46.948 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "default", 2024-09-15T20:28:46.948 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:28:46.949 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "root", 2024-09-15T20:28:46.949 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:28:46.949 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:28:46.949 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:28:46.949 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:28:46.949 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.949 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -3, 2024-09-15T20:28:46.949 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:28:46.949 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:28:46.949 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.949 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.949 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.949 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.949 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -2, 2024-09-15T20:28:46.950 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "default~hdd", 2024-09-15T20:28:46.950 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:28:46.950 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "root", 2024-09-15T20:28:46.950 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:28:46.950 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:28:46.950 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:28:46.950 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:28:46.950 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.950 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -4, 2024-09-15T20:28:46.950 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:28:46.950 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:28:46.950 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.950 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.950 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.950 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.951 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -3, 2024-09-15T20:28:46.951 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "smithi139", 2024-09-15T20:28:46.951 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:28:46.951 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "host", 2024-09-15T20:28:46.951 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:28:46.951 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:28:46.951 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:28:46.951 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:28:46.951 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.951 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:28:46.951 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:28:46.951 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:28:46.951 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.951 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.952 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:28:46.952 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:28:46.952 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 1 2024-09-15T20:28:46.952 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.952 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.952 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:28:46.952 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:28:46.952 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 2 2024-09-15T20:28:46.952 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.952 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.952 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.952 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.952 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -4, 2024-09-15T20:28:46.952 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "smithi139~hdd", 2024-09-15T20:28:46.953 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:28:46.953 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "host", 2024-09-15T20:28:46.953 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:28:46.953 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:28:46.953 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:28:46.953 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:28:46.953 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.953 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:28:46.953 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:28:46.953 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:28:46.953 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.953 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.953 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:28:46.954 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:28:46.954 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 1 2024-09-15T20:28:46.954 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.954 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.954 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:28:46.954 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:28:46.954 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 2 2024-09-15T20:28:46.954 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.954 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.954 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.954 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:28:46.954 INFO:tasks.workunit.client.0.smithi139.stdout: "rules": [ 2024-09-15T20:28:46.954 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.955 INFO:tasks.workunit.client.0.smithi139.stdout: "rule_id": 0, 2024-09-15T20:28:46.955 INFO:tasks.workunit.client.0.smithi139.stdout: "rule_name": "replicated_rule", 2024-09-15T20:28:46.955 INFO:tasks.workunit.client.0.smithi139.stdout: "type": 1, 2024-09-15T20:28:46.955 INFO:tasks.workunit.client.0.smithi139.stdout: "steps": [ 2024-09-15T20:28:46.955 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.955 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "take", 2024-09-15T20:28:46.955 INFO:tasks.workunit.client.0.smithi139.stdout: "item": -1, 2024-09-15T20:28:46.955 INFO:tasks.workunit.client.0.smithi139.stdout: "item_name": "default" 2024-09-15T20:28:46.955 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.955 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.955 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "choose_firstn", 2024-09-15T20:28:46.955 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 0, 2024-09-15T20:28:46.955 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "osd" 2024-09-15T20:28:46.955 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.955 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.955 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "emit" 2024-09-15T20:28:46.956 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.956 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:28:46.956 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.956 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:28:46.956 INFO:tasks.workunit.client.0.smithi139.stdout: "tunables": { 2024-09-15T20:28:46.956 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_local_tries": 0, 2024-09-15T20:28:46.956 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_local_fallback_tries": 0, 2024-09-15T20:28:46.956 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_total_tries": 50, 2024-09-15T20:28:46.956 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_descend_once": 1, 2024-09-15T20:28:46.956 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_vary_r": 1, 2024-09-15T20:28:46.956 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_stable": 1, 2024-09-15T20:28:46.956 INFO:tasks.workunit.client.0.smithi139.stdout: "msr_descents": 100, 2024-09-15T20:28:46.956 INFO:tasks.workunit.client.0.smithi139.stdout: "msr_collision_tries": 100, 2024-09-15T20:28:46.956 INFO:tasks.workunit.client.0.smithi139.stdout: "straw_calc_version": 1, 2024-09-15T20:28:46.956 INFO:tasks.workunit.client.0.smithi139.stdout: "allowed_bucket_algs": 54, 2024-09-15T20:28:46.957 INFO:tasks.workunit.client.0.smithi139.stdout: "profile": "jewel", 2024-09-15T20:28:46.957 INFO:tasks.workunit.client.0.smithi139.stdout: "optimal_tunables": 1, 2024-09-15T20:28:46.957 INFO:tasks.workunit.client.0.smithi139.stdout: "legacy_tunables": 0, 2024-09-15T20:28:46.957 INFO:tasks.workunit.client.0.smithi139.stdout: "minimum_required_version": "jewel", 2024-09-15T20:28:46.957 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables": 1, 2024-09-15T20:28:46.957 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables2": 1, 2024-09-15T20:28:46.957 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v2_rules": 0, 2024-09-15T20:28:46.957 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables3": 1, 2024-09-15T20:28:46.957 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v3_rules": 0, 2024-09-15T20:28:46.957 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v4_buckets": 1, 2024-09-15T20:28:46.957 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables5": 1, 2024-09-15T20:28:46.957 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v5_rules": 0, 2024-09-15T20:28:46.957 INFO:tasks.workunit.client.0.smithi139.stdout: "has_msr_rules": 0 2024-09-15T20:28:46.957 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.958 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_args": {} 2024-09-15T20:28:46.958 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.958 INFO:tasks.workunit.client.0.smithi139.stdout: "fsmap": { 2024-09-15T20:28:46.958 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 1, 2024-09-15T20:28:46.958 INFO:tasks.workunit.client.0.smithi139.stdout: "btime": "2024-09-15T20:25:18:543398+0000", 2024-09-15T20:28:46.958 INFO:tasks.workunit.client.0.smithi139.stdout: "default_fscid": -1, 2024-09-15T20:28:46.958 INFO:tasks.workunit.client.0.smithi139.stdout: "compat": { 2024-09-15T20:28:46.958 INFO:tasks.workunit.client.0.smithi139.stdout: "compat": {}, 2024-09-15T20:28:46.958 INFO:tasks.workunit.client.0.smithi139.stdout: "ro_compat": {}, 2024-09-15T20:28:46.958 INFO:tasks.workunit.client.0.smithi139.stdout: "incompat": { 2024-09-15T20:28:46.958 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_1": "base v0.20", 2024-09-15T20:28:46.958 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_2": "client writeable ranges", 2024-09-15T20:28:46.958 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_3": "default file layouts on dirs", 2024-09-15T20:28:46.958 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_4": "dir inode in separate object", 2024-09-15T20:28:46.958 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_5": "mds uses versioned encoding", 2024-09-15T20:28:46.959 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-15T20:28:46.959 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_8": "no anchor table", 2024-09-15T20:28:46.959 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_9": "file layout v2", 2024-09-15T20:28:46.959 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_10": "snaprealm v2", 2024-09-15T20:28:46.959 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_11": "minor log segments", 2024-09-15T20:28:46.959 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_12": "quiesce subvolumes" 2024-09-15T20:28:46.959 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.959 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.959 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_flags": { 2024-09-15T20:28:46.959 INFO:tasks.workunit.client.0.smithi139.stdout: "enable_multiple": true, 2024-09-15T20:28:46.959 INFO:tasks.workunit.client.0.smithi139.stdout: "ever_enabled_multiple": true 2024-09-15T20:28:46.959 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.959 INFO:tasks.workunit.client.0.smithi139.stdout: "standbys": [], 2024-09-15T20:28:46.959 INFO:tasks.workunit.client.0.smithi139.stdout: "filesystems": [] 2024-09-15T20:28:46.959 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.960 INFO:tasks.workunit.client.0.smithi139.stdout: "mdsmap_first_committed": 1, 2024-09-15T20:28:46.960 INFO:tasks.workunit.client.0.smithi139.stdout: "mdsmap_last_committed": 1, 2024-09-15T20:28:46.960 INFO:tasks.workunit.client.0.smithi139.stdout: "auth": { 2024-09-15T20:28:46.960 INFO:tasks.workunit.client.0.smithi139.stdout: "first_committed": 1, 2024-09-15T20:28:46.960 INFO:tasks.workunit.client.0.smithi139.stdout: "last_committed": 5, 2024-09-15T20:28:46.960 INFO:tasks.workunit.client.0.smithi139.stdout: "num_secrets": 3 2024-09-15T20:28:46.960 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.960 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg": 1, 2024-09-15T20:28:46.960 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_active": 1, 2024-09-15T20:28:46.960 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_unknown": 0, 2024-09-15T20:28:46.960 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osd": 3, 2024-09-15T20:28:46.960 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_sum": { 2024-09-15T20:28:46.960 INFO:tasks.workunit.client.0.smithi139.stdout: "stat_sum": { 2024-09-15T20:28:46.960 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes": 61700, 2024-09-15T20:28:46.961 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects": 100, 2024-09-15T20:28:46.961 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_clones": 0, 2024-09-15T20:28:46.961 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_copies": 300, 2024-09-15T20:28:46.961 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T20:28:46.961 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing": 0, 2024-09-15T20:28:46.961 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_degraded": 100, 2024-09-15T20:28:46.961 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_misplaced": 0, 2024-09-15T20:28:46.961 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_unfound": 0, 2024-09-15T20:28:46.961 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_dirty": 100, 2024-09-15T20:28:46.961 INFO:tasks.workunit.client.0.smithi139.stdout: "num_whiteouts": 0, 2024-09-15T20:28:46.961 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read": 0, 2024-09-15T20:28:46.961 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read_kb": 0, 2024-09-15T20:28:46.961 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write": 100, 2024-09-15T20:28:46.961 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write_kb": 100, 2024-09-15T20:28:46.962 INFO:tasks.workunit.client.0.smithi139.stdout: "num_scrub_errors": 0, 2024-09-15T20:28:46.962 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T20:28:46.962 INFO:tasks.workunit.client.0.smithi139.stdout: "num_deep_scrub_errors": 0, 2024-09-15T20:28:46.962 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_recovered": 0, 2024-09-15T20:28:46.962 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_recovered": 0, 2024-09-15T20:28:46.962 INFO:tasks.workunit.client.0.smithi139.stdout: "num_keys_recovered": 0, 2024-09-15T20:28:46.962 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_omap": 0, 2024-09-15T20:28:46.962 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T20:28:46.962 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T20:28:46.962 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush": 0, 2024-09-15T20:28:46.962 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_kb": 0, 2024-09-15T20:28:46.962 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict": 0, 2024-09-15T20:28:46.962 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_kb": 0, 2024-09-15T20:28:46.962 INFO:tasks.workunit.client.0.smithi139.stdout: "num_promote": 0, 2024-09-15T20:28:46.962 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_high": 0, 2024-09-15T20:28:46.962 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_low": 0, 2024-09-15T20:28:46.963 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_some": 0, 2024-09-15T20:28:46.963 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_full": 0, 2024-09-15T20:28:46.963 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_pinned": 0, 2024-09-15T20:28:46.963 INFO:tasks.workunit.client.0.smithi139.stdout: "num_legacy_snapsets": 0, 2024-09-15T20:28:46.963 INFO:tasks.workunit.client.0.smithi139.stdout: "num_large_omap_objects": 0, 2024-09-15T20:28:46.963 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_manifest": 0, 2024-09-15T20:28:46.963 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_bytes": 0, 2024-09-15T20:28:46.963 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_keys": 0, 2024-09-15T20:28:46.963 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_repaired": 0 2024-09-15T20:28:46.963 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.963 INFO:tasks.workunit.client.0.smithi139.stdout: "store_stats": { 2024-09-15T20:28:46.963 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 0, 2024-09-15T20:28:46.963 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 0, 2024-09-15T20:28:46.964 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:28:46.964 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 0, 2024-09-15T20:28:46.964 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 0, 2024-09-15T20:28:46.964 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:28:46.964 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:28:46.964 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:28:46.964 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 0, 2024-09-15T20:28:46.964 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 0 2024-09-15T20:28:46.964 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.964 INFO:tasks.workunit.client.0.smithi139.stdout: "log_size": 16, 2024-09-15T20:28:46.964 INFO:tasks.workunit.client.0.smithi139.stdout: "ondisk_log_size": 16, 2024-09-15T20:28:46.964 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 2, 2024-09-15T20:28:46.964 INFO:tasks.workunit.client.0.smithi139.stdout: "acting": 2, 2024-09-15T20:28:46.965 INFO:tasks.workunit.client.0.smithi139.stdout: "num_store_stats": 0 2024-09-15T20:28:46.965 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.965 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_sum": { 2024-09-15T20:28:46.965 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 0, 2024-09-15T20:28:46.965 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 0, 2024-09-15T20:28:46.965 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pgs": 3, 2024-09-15T20:28:46.965 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osds": 3, 2024-09-15T20:28:46.965 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_osds": 3, 2024-09-15T20:28:46.965 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T20:28:46.965 INFO:tasks.workunit.client.0.smithi139.stdout: "kb": 314572800, 2024-09-15T20:28:46.965 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used": 83664, 2024-09-15T20:28:46.965 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_data": 2112, 2024-09-15T20:28:46.965 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_omap": 10, 2024-09-15T20:28:46.965 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_meta": 81461, 2024-09-15T20:28:46.965 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_avail": 314489136, 2024-09-15T20:28:46.966 INFO:tasks.workunit.client.0.smithi139.stdout: "statfs": { 2024-09-15T20:28:46.966 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 322122547200, 2024-09-15T20:28:46.966 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 322036875264, 2024-09-15T20:28:46.966 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:28:46.966 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 2162688, 2024-09-15T20:28:46.966 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 460132, 2024-09-15T20:28:46.966 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:28:46.966 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:28:46.966 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:28:46.966 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 10396, 2024-09-15T20:28:46.966 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 83416932 2024-09-15T20:28:46.966 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.966 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_peers": [], 2024-09-15T20:28:46.966 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_trim_queue_len": 0, 2024-09-15T20:28:46.966 INFO:tasks.workunit.client.0.smithi139.stdout: "num_snap_trimming": 0, 2024-09-15T20:28:46.967 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shards_repaired": 1, 2024-09-15T20:28:46.967 INFO:tasks.workunit.client.0.smithi139.stdout: "op_queue_age_hist": { 2024-09-15T20:28:46.967 INFO:tasks.workunit.client.0.smithi139.stdout: "histogram": [], 2024-09-15T20:28:46.967 INFO:tasks.workunit.client.0.smithi139.stdout: "upper_bound": 1 2024-09-15T20:28:46.967 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.967 INFO:tasks.workunit.client.0.smithi139.stdout: "perf_stat": { 2024-09-15T20:28:46.967 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ms": 0, 2024-09-15T20:28:46.967 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ms": 0, 2024-09-15T20:28:46.967 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ns": 0, 2024-09-15T20:28:46.967 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ns": 0 2024-09-15T20:28:46.967 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.967 INFO:tasks.workunit.client.0.smithi139.stdout: "alerts": [], 2024-09-15T20:28:46.967 INFO:tasks.workunit.client.0.smithi139.stdout: "network_ping_times": [] 2024-09-15T20:28:46.967 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.968 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_sum_by_class": { 2024-09-15T20:28:46.968 INFO:tasks.workunit.client.0.smithi139.stdout: "hdd": { 2024-09-15T20:28:46.968 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 0, 2024-09-15T20:28:46.968 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 0, 2024-09-15T20:28:46.968 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pgs": 3, 2024-09-15T20:28:46.968 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osds": 3, 2024-09-15T20:28:46.968 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_osds": 3, 2024-09-15T20:28:46.968 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T20:28:46.968 INFO:tasks.workunit.client.0.smithi139.stdout: "kb": 314572800, 2024-09-15T20:28:46.968 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used": 83664, 2024-09-15T20:28:46.968 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_data": 2112, 2024-09-15T20:28:46.968 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_omap": 10, 2024-09-15T20:28:46.968 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_meta": 81461, 2024-09-15T20:28:46.968 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_avail": 314489136, 2024-09-15T20:28:46.968 INFO:tasks.workunit.client.0.smithi139.stdout: "statfs": { 2024-09-15T20:28:46.968 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 322122547200, 2024-09-15T20:28:46.969 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 322036875264, 2024-09-15T20:28:46.969 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:28:46.969 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 2162688, 2024-09-15T20:28:46.969 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 460132, 2024-09-15T20:28:46.969 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:28:46.969 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:28:46.969 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:28:46.969 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 10396, 2024-09-15T20:28:46.969 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 83416932 2024-09-15T20:28:46.969 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.969 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_peers": [], 2024-09-15T20:28:46.969 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_trim_queue_len": 0, 2024-09-15T20:28:46.969 INFO:tasks.workunit.client.0.smithi139.stdout: "num_snap_trimming": 0, 2024-09-15T20:28:46.969 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shards_repaired": 1, 2024-09-15T20:28:46.969 INFO:tasks.workunit.client.0.smithi139.stdout: "op_queue_age_hist": { 2024-09-15T20:28:46.969 INFO:tasks.workunit.client.0.smithi139.stdout: "histogram": [], 2024-09-15T20:28:46.970 INFO:tasks.workunit.client.0.smithi139.stdout: "upper_bound": 1 2024-09-15T20:28:46.970 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.970 INFO:tasks.workunit.client.0.smithi139.stdout: "perf_stat": { 2024-09-15T20:28:46.970 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ms": 0, 2024-09-15T20:28:46.970 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ms": 0, 2024-09-15T20:28:46.970 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ns": 0, 2024-09-15T20:28:46.970 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ns": 0 2024-09-15T20:28:46.970 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.970 INFO:tasks.workunit.client.0.smithi139.stdout: "alerts": [], 2024-09-15T20:28:46.970 INFO:tasks.workunit.client.0.smithi139.stdout: "network_ping_times": [] 2024-09-15T20:28:46.970 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.970 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.970 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_stats": [ 2024-09-15T20:28:46.970 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.971 INFO:tasks.workunit.client.0.smithi139.stdout: "poolid": 1, 2024-09-15T20:28:46.971 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg": 1, 2024-09-15T20:28:46.971 INFO:tasks.workunit.client.0.smithi139.stdout: "stat_sum": { 2024-09-15T20:28:46.971 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes": 61700, 2024-09-15T20:28:46.971 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects": 100, 2024-09-15T20:28:46.971 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_clones": 0, 2024-09-15T20:28:46.971 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_copies": 300, 2024-09-15T20:28:46.971 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T20:28:46.971 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing": 0, 2024-09-15T20:28:46.971 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_degraded": 100, 2024-09-15T20:28:46.971 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_misplaced": 0, 2024-09-15T20:28:46.971 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_unfound": 0, 2024-09-15T20:28:46.971 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_dirty": 100, 2024-09-15T20:28:46.971 INFO:tasks.workunit.client.0.smithi139.stdout: "num_whiteouts": 0, 2024-09-15T20:28:46.971 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read": 0, 2024-09-15T20:28:46.972 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read_kb": 0, 2024-09-15T20:28:46.972 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write": 100, 2024-09-15T20:28:46.972 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write_kb": 100, 2024-09-15T20:28:46.972 INFO:tasks.workunit.client.0.smithi139.stdout: "num_scrub_errors": 0, 2024-09-15T20:28:46.972 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T20:28:46.972 INFO:tasks.workunit.client.0.smithi139.stdout: "num_deep_scrub_errors": 0, 2024-09-15T20:28:46.972 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_recovered": 0, 2024-09-15T20:28:46.972 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_recovered": 0, 2024-09-15T20:28:46.972 INFO:tasks.workunit.client.0.smithi139.stdout: "num_keys_recovered": 0, 2024-09-15T20:28:46.972 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_omap": 0, 2024-09-15T20:28:46.972 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T20:28:46.972 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T20:28:46.972 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush": 0, 2024-09-15T20:28:46.973 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_kb": 0, 2024-09-15T20:28:46.973 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict": 0, 2024-09-15T20:28:46.973 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_kb": 0, 2024-09-15T20:28:46.973 INFO:tasks.workunit.client.0.smithi139.stdout: "num_promote": 0, 2024-09-15T20:28:46.973 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_high": 0, 2024-09-15T20:28:46.973 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_low": 0, 2024-09-15T20:28:46.973 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_some": 0, 2024-09-15T20:28:46.973 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_full": 0, 2024-09-15T20:28:46.973 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_pinned": 0, 2024-09-15T20:28:46.973 INFO:tasks.workunit.client.0.smithi139.stdout: "num_legacy_snapsets": 0, 2024-09-15T20:28:46.973 INFO:tasks.workunit.client.0.smithi139.stdout: "num_large_omap_objects": 0, 2024-09-15T20:28:46.973 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_manifest": 0, 2024-09-15T20:28:46.973 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_bytes": 0, 2024-09-15T20:28:46.974 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_keys": 0, 2024-09-15T20:28:46.974 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_repaired": 0 2024-09-15T20:28:46.974 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.974 INFO:tasks.workunit.client.0.smithi139.stdout: "store_stats": { 2024-09-15T20:28:46.974 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 0, 2024-09-15T20:28:46.974 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 0, 2024-09-15T20:28:46.974 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:28:46.974 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 1228800, 2024-09-15T20:28:46.974 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 185100, 2024-09-15T20:28:46.974 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:28:46.974 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:28:46.974 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:28:46.975 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 0, 2024-09-15T20:28:46.975 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 0 2024-09-15T20:28:46.975 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.975 INFO:tasks.workunit.client.0.smithi139.stdout: "log_size": 16, 2024-09-15T20:28:46.975 INFO:tasks.workunit.client.0.smithi139.stdout: "ondisk_log_size": 16, 2024-09-15T20:28:46.975 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 2, 2024-09-15T20:28:46.975 INFO:tasks.workunit.client.0.smithi139.stdout: "acting": 2, 2024-09-15T20:28:46.975 INFO:tasks.workunit.client.0.smithi139.stdout: "num_store_stats": 3 2024-09-15T20:28:46.975 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.975 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:28:46.976 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_stats": [ 2024-09-15T20:28:46.976 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.976 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:28:46.976 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 154618822667 2024-09-15T20:28:46.976 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.976 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.976 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:28:46.976 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 42949672989 2024-09-15T20:28:46.976 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.976 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.976 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:28:46.976 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 163208757258 2024-09-15T20:28:46.976 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.977 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:28:46.977 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_by_state": [ 2024-09-15T20:28:46.977 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.977 INFO:tasks.workunit.client.0.smithi139.stdout: "state": "active+undersized+degraded", 2024-09-15T20:28:46.977 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 1 2024-09-15T20:28:46.977 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.977 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:28:46.977 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_by_osd": [ 2024-09-15T20:28:46.977 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.977 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:28:46.977 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 0, 2024-09-15T20:28:46.977 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 1, 2024-09-15T20:28:46.978 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:28:46.978 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.978 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.978 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:28:46.978 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 1, 2024-09-15T20:28:46.978 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 1, 2024-09-15T20:28:46.978 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:28:46.978 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.978 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.978 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:28:46.978 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 0, 2024-09-15T20:28:46.978 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 0, 2024-09-15T20:28:46.978 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:28:46.978 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.979 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:28:46.979 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snaps": [ 2024-09-15T20:28:46.979 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:28:46.979 INFO:tasks.workunit.client.0.smithi139.stdout: "pool": 1, 2024-09-15T20:28:46.979 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snaps": {} 2024-09-15T20:28:46.979 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.979 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:28:46.979 INFO:tasks.workunit.client.0.smithi139.stdout: "servicemap": { 2024-09-15T20:28:46.979 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 5, 2024-09-15T20:28:46.979 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:28:06.168649+0000", 2024-09-15T20:28:46.979 INFO:tasks.workunit.client.0.smithi139.stdout: "services": { 2024-09-15T20:28:46.979 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": { 2024-09-15T20:28:46.979 INFO:tasks.workunit.client.0.smithi139.stdout: "daemons": { 2024-09-15T20:28:46.979 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": "", 2024-09-15T20:28:46.979 INFO:tasks.workunit.client.0.smithi139.stdout: "0": { 2024-09-15T20:28:46.980 INFO:tasks.workunit.client.0.smithi139.stdout: "start_epoch": 0, 2024-09-15T20:28:46.980 INFO:tasks.workunit.client.0.smithi139.stdout: "start_stamp": "0.000000", 2024-09-15T20:28:46.980 INFO:tasks.workunit.client.0.smithi139.stdout: "gid": 0, 2024-09-15T20:28:46.980 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "(unrecognized address family 0)/0", 2024-09-15T20:28:46.980 INFO:tasks.workunit.client.0.smithi139.stdout: "metadata": {}, 2024-09-15T20:28:46.980 INFO:tasks.workunit.client.0.smithi139.stdout: "task_status": {} 2024-09-15T20:28:46.980 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.980 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.980 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.980 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.980 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:28:46.980 INFO:tasks.workunit.client.0.smithi139.stdout: "mgrstat_first_committed": 1, 2024-09-15T20:28:46.980 INFO:tasks.workunit.client.0.smithi139.stdout: "mgrstat_last_committed": 93, 2024-09-15T20:28:46.980 INFO:tasks.workunit.client.0.smithi139.stdout: "logm_first_committed": 1, 2024-09-15T20:28:46.981 INFO:tasks.workunit.client.0.smithi139.stdout: "logm_last_committed": 382, 2024-09-15T20:28:46.981 INFO:tasks.workunit.client.0.smithi139.stdout: "paxos": { 2024-09-15T20:28:46.981 INFO:tasks.workunit.client.0.smithi139.stdout: "first_committed": 1, 2024-09-15T20:28:46.981 INFO:tasks.workunit.client.0.smithi139.stdout: "last_committed": 526, 2024-09-15T20:28:46.981 INFO:tasks.workunit.client.0.smithi139.stdout: "last_pn": 0, 2024-09-15T20:28:46.981 INFO:tasks.workunit.client.0.smithi139.stdout: "accepted_pn": 0 2024-09-15T20:28:46.981 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:28:46.981 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:28:46.982 INFO:tasks.workunit.client.0.smithi139.stderr:report 3157982189 2024-09-15T20:28:46.982 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-15T20:28:46.982 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2024-09-15T20:28:46.982 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2024-09-15T20:28:46.982 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2024-09-15T20:28:46.982 INFO:tasks.workunit.client.0.smithi139.stdout:writing non-divergent object existing_4 2024-09-15T20:28:46.982 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2024-09-15T20:28:47.295 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:28:47.295 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-15T20:28:09.023943+0000 29'100 41:133 [0,2] 0 [0,2] 0 0'0 2024-09-15T20:27:08.062076+0000 0'0 2024-09-15T20:27:08.062076+0000 0 0 periodic scrub scheduled @ 2024-09-16T21:54:37.328034+0000 0 0 2024-09-15T20:28:47.295 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:28:47.295 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:28:47.296 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:28:47.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2024-09-15T20:28:47.405 INFO:tasks.workunit.client.0.smithi139.stdout:delay recovery 2024-09-15T20:28:47.405 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2024-09-15T20:28:47.405 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2024-09-15T20:28:47.825 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:28:47.825 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-15T20:28:09.023943+0000 29'100 41:133 [0,2] 0 [0,2] 0 0'0 2024-09-15T20:27:08.062076+0000 0'0 2024-09-15T20:27:08.062076+0000 0 0 periodic scrub scheduled @ 2024-09-16T21:54:37.328034+0000 0 0 2024-09-15T20:28:47.825 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:28:47.825 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:28:47.825 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:28:47.843 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-09-15T20:28:47.844 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2024-09-15T20:28:47.844 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T20:28:47.844 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T20:28:47.844 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:28:47.844 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:28:47.845 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:28:47.845 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.0.asok 2024-09-15T20:28:47.845 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-09-15T20:28:47.845 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.0.asok set_recovery_delay 100000 2024-09-15T20:28:47.964 INFO:tasks.workunit.client.0.smithi139.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T20:28:47.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-09-15T20:28:47.976 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2024-09-15T20:28:47.976 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-15T20:28:47.976 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-15T20:28:47.977 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:28:47.977 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:28:47.977 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:28:47.977 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.2.asok 2024-09-15T20:28:47.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-09-15T20:28:47.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.2.asok set_recovery_delay 100000 2024-09-15T20:28:48.077 INFO:tasks.workunit.client.0.smithi139.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T20:28:48.088 INFO:tasks.workunit.client.0.smithi139.stdout:revive divergent 1 2024-09-15T20:28:48.089 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2024-09-15T20:28:48.089 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2024-09-15T20:28:48.498 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:28:48.499 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-15T20:28:09.023943+0000 29'100 41:133 [0,2] 0 [0,2] 0 0'0 2024-09-15T20:27:08.062076+0000 0'0 2024-09-15T20:27:08.062076+0000 0 0 periodic scrub scheduled @ 2024-09-16T21:54:37.328034+0000 0 0 2024-09-15T20:28:48.499 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:28:48.499 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:28:48.499 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:28:48.514 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2024-09-15T20:28:49.146 INFO:tasks.workunit.client.0.smithi139.stderr:noup is set 2024-09-15T20:28:49.168 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2024-09-15T20:28:49.168 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-15T20:28:49.168 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:28:49.168 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:28:49.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:28:49.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-15T20:28:49.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:28:49.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:28:49.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:28:49.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:28:49.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T20:28:49.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T20:28:49.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:28:49.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:28:49.170 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:28:49.170 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:28:49.170 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:28:49.170 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:28:49.170 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:28:49.170 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:28:49.170 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:28:49.173 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:28:49.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:28:49.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:28:49.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:28:49.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:28:49.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:28:49.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:28:49.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:28:49.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:28:49.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:28:49.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:28:49.174 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-15T20:28:49.174 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.1 2024-09-15T20:28:49.175 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:28:49.175 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:28:49.175 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-15T20:28:49.175 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:28:49.175 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:28:49.175 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:28:49.175 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:28:49.216 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:28:49.211+0000 7fc9b755f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:49.218 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:28:49.215+0000 7fc9b755f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:49.220 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:28:49.215+0000 7fc9b755f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:49.667 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-15T20:28:49.668 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2024-09-15T20:28:50.701 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:28:50.695+0000 7fc9b755f7c0 -1 Falling back to public interface 2024-09-15T20:28:53.840 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:28:53.835+0000 7fc9b755f7c0 -1 osd.1 29 log_to_monitors true 2024-09-15T20:28:54.669 INFO:tasks.workunit.client.0.smithi139.stdout:delay recovery divergent 2024-09-15T20:28:54.669 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2024-09-15T20:28:54.670 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2024-09-15T20:28:55.109 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:28:55.109 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-15T20:28:09.023943+0000 41'101 43:138 [0,2] 0 [0,2] 0 0'0 2024-09-15T20:27:08.062076+0000 0'0 2024-09-15T20:27:08.062076+0000 0 0 periodic scrub scheduled @ 2024-09-16T21:54:37.328034+0000 0 0 2024-09-15T20:28:55.109 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:28:55.110 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:28:55.110 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:28:55.128 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2024-09-15T20:28:55.128 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T20:28:55.128 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T20:28:55.128 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:28:55.128 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:28:55.129 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:28:55.129 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.1.asok 2024-09-15T20:28:55.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2024-09-15T20:28:55.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.1.asok set_recovery_delay 100000 2024-09-15T20:28:55.242 INFO:tasks.workunit.client.0.smithi139.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T20:28:55.254 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2024-09-15T20:28:55.967 INFO:tasks.workunit.client.0.smithi139.stderr:noup is unset 2024-09-15T20:28:55.986 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2024-09-15T20:28:55.986 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:28:55.986 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:28:55.986 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:28:55.986 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:28:55.986 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:28:55.986 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:28:55.986 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:28:55.987 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:28:55.987 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:28:56.625 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/3285624630,v1:127.0.0.1:6803/3285624630] [v2:127.0.0.1:6804/3285624630,v1:127.0.0.1:6805/3285624630] exists,up cdf05e2d-6e34-45ea-89d8-b5b29cbb7f44 2024-09-15T20:28:56.626 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:28:56.626 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:28:56.627 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:28:56.627 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2024-09-15T20:28:56.627 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:28:56.627 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:28:56.627 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:28:56.627 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:28:56.627 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:28:56.627 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:28:56.628 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:28:56.628 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:28:56.628 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:28:57.100 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/1751055914,v1:127.0.0.1:6819/1751055914] [v2:127.0.0.1:6820/1751055914,v1:127.0.0.1:6821/1751055914] exists,up 95c45dfa-5b44-4aaf-9fc1-ebb69bbe1237 2024-09-15T20:28:57.102 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:28:57.102 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:28:57.102 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:28:57.102 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2024-09-15T20:28:57.102 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:28:57.102 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:28:57.102 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:28:57.102 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:28:57.102 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:28:57.102 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:28:57.103 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:28:57.103 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:28:57.103 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:28:57.536 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/919482650,v1:127.0.0.1:6811/919482650] [v2:127.0.0.1:6812/919482650,v1:127.0.0.1:6813/919482650] exists,up 9d0447d3-39ad-48fb-8e36-2d8f2ba5c93b 2024-09-15T20:28:57.536 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:28:57.537 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:28:57.537 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:28:57.537 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2024-09-15T20:28:57.979 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:28:57.979 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-15T20:28:09.023943+0000 41'101 43:138 [0,2] 0 [0,2] 0 0'0 2024-09-15T20:27:08.062076+0000 0'0 2024-09-15T20:27:08.062076+0000 0 0 periodic scrub scheduled @ 2024-09-16T21:54:37.328034+0000 0 0 2024-09-15T20:28:57.980 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:28:57.980 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:28:57.980 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:28:57.996 INFO:tasks.workunit.client.0.smithi139.stdout:wait for peering 2024-09-15T20:28:57.997 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2024-09-15T20:28:57.997 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2024-09-15T20:28:58.409 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:28:58.409 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-15T20:28:09.023943+0000 41'101 43:138 [0,2] 0 [0,2] 0 0'0 2024-09-15T20:27:08.062076+0000 0'0 2024-09-15T20:27:08.062076+0000 0 0 periodic scrub scheduled @ 2024-09-16T21:54:37.328034+0000 0 0 2024-09-15T20:28:58.409 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:28:58.409 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:28:58.409 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:28:58.426 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2024-09-15T20:28:58.570 INFO:tasks.workunit.client.0.smithi139.stdout:killing divergent 1 2024-09-15T20:28:58.570 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2024-09-15T20:28:58.570 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2024-09-15T20:28:58.977 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:28:58.977 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-15T20:28:09.023943+0000 41'101 43:138 [0,2] 0 [0,2] 0 0'0 2024-09-15T20:27:08.062076+0000 0'0 2024-09-15T20:27:08.062076+0000 0 0 periodic scrub scheduled @ 2024-09-16T21:54:37.328034+0000 0 0 2024-09-15T20:28:58.977 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:28:58.977 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:28:58.978 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:28:58.993 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2024-09-15T20:28:58.994 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:28:58.994 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:28:58.994 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:28:58.994 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:28:58.994 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:28:59.101 INFO:tasks.workunit.client.0.smithi139.stdout:reviving divergent 1 2024-09-15T20:28:59.102 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:28:59.102 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2024-09-15T20:28:59.102 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2024-09-15T20:28:59.504 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:28:59.504 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-15T20:28:09.023943+0000 41'101 43:138 [0,2] 0 [0,2] 0 0'0 2024-09-15T20:27:08.062076+0000 0'0 2024-09-15T20:27:08.062076+0000 0 0 periodic scrub scheduled @ 2024-09-16T21:54:37.328034+0000 0 0 2024-09-15T20:28:59.504 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:28:59.504 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:28:59.504 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:28:59.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2024-09-15T20:28:59.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-15T20:28:59.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:28:59.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:28:59.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:28:59.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-15T20:28:59.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:28:59.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:28:59.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:28:59.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:28:59.522 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T20:28:59.522 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T20:28:59.522 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:28:59.522 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:28:59.522 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:28:59.522 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:28:59.522 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:28:59.522 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:28:59.523 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:28:59.524 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:28:59.524 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:28:59.524 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:28:59.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:28:59.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:28:59.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:28:59.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:28:59.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:28:59.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:28:59.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:28:59.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:28:59.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:28:59.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:28:59.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-15T20:28:59.525 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:28:59.525 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.1 2024-09-15T20:28:59.526 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:28:59.526 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-15T20:28:59.526 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:28:59.526 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:28:59.526 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:28:59.526 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:28:59.559 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:28:59.555+0000 7f09cfc107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:59.561 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:28:59.555+0000 7f09cfc107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:28:59.562 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:28:59.559+0000 7f09cfc107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:29:00.091 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:29:00.091 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:29:00.091 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:29:00.091 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:29:00.091 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:29:00.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:29:00.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:29:00.092 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:29:00.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:29:00.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:29:00.540 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/1751055914,v1:127.0.0.1:6819/1751055914] [v2:127.0.0.1:6820/1751055914,v1:127.0.0.1:6821/1751055914] exists,up 95c45dfa-5b44-4aaf-9fc1-ebb69bbe1237 2024-09-15T20:29:00.540 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:29:00.540 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:29:00.541 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:29:00.541 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2024-09-15T20:29:00.599 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:29:00.595+0000 7f09cfc107c0 -1 Falling back to public interface 2024-09-15T20:29:03.850 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:29:03.842+0000 7f09cfc107c0 -1 osd.1 47 log_to_monitors true 2024-09-15T20:29:20.541 INFO:tasks.workunit.client.0.smithi139.stdout:allowing recovery 2024-09-15T20:29:20.542 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2024-09-15T20:29:20.542 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2024-09-15T20:29:20.999 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:29:20.999 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-09-15T20:29:05.558743+0000 47'102 51:161 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:27:08.062076+0000 0'0 2024-09-15T20:27:08.062076+0000 0 0 periodic scrub scheduled @ 2024-09-17T01:06:20.405437+0000 0 0 2024-09-15T20:29:20.999 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:29:20.999 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:29:21.000 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:29:21.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-15T20:29:21.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-15T20:29:21.169 INFO:tasks.workunit.client.0.smithi139.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T20:29:21.183 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-15T20:29:21.183 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-15T20:29:21.332 INFO:tasks.workunit.client.0.smithi139.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T20:29:21.346 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-15T20:29:21.346 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-15T20:29:21.494 INFO:tasks.workunit.client.0.smithi139.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T20:29:21.507 INFO:tasks.workunit.client.0.smithi139.stdout:reading divergent objects 2024-09-15T20:29:21.507 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2024-09-15T20:29:21.507 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2024-09-15T20:29:21.925 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:29:21.925 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-09-15T20:29:05.558743+0000 47'102 51:161 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:27:08.062076+0000 0'0 2024-09-15T20:27:08.062076+0000 0 0 periodic scrub scheduled @ 2024-09-17T01:06:20.405437+0000 0 0 2024-09-15T20:29:21.925 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:29:21.925 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:29:21.926 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:29:21.941 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2024-09-15T20:29:21.942 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2024-09-15T20:29:21.943 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T20:29:21.943 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2024-09-15T20:29:21.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T20:29:21.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2024-09-15T20:29:22.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T20:29:22.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2024-09-15T20:29:22.048 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T20:29:22.048 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2024-09-15T20:29:22.081 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2024-09-15T20:29:22.082 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-09-15T20:29:22.083 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-15T20:29:22.088 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:28:57.367+0000 7fc99b4e9640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4620.0:1 2024-09-15T20:27:42.089198+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-09-15T20:29:22.088 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:28:57.367+0000 7fc99b4e9640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4620.0:1 2024-09-15T20:27:42.089198+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-09-15T20:29:22.089 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:28:57.367+0000 7fc99b4e9640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T20:29:22.089 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:28:57.367+0000 7fc99b4e9640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-09-15T20:29:22.089 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:28:57.367+0000 7fc99b4e9640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-09-15T20:29:22.089 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:28:57.367+0000 7fc99b4e9640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-09-15T20:29:22.089 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:28:57.367+0000 7fc99b4e9640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2024-09-15T20:29:22.089 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:28:57.367+0000 7fc99b4e9640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4620.0:1 2024-09-15T20:27:42.089198+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-09-15T20:29:22.089 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:28:57.367+0000 7fc99b4e9640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-09-15T20:29:22.089 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-09-15T20:29:22.090 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-15T20:29:22.094 INFO:tasks.workunit.client.0.smithi139.stdout:success 2024-09-15T20:29:22.094 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2024-09-15T20:29:22.094 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2024-09-15T20:29:22.094 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-15T20:29:22.095 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-15T20:29:22.599 INFO:tasks.workunit.client.0.smithi139.stderr:pool 'test' does not exist 2024-09-15T20:29:22.615 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2024-09-15T20:29:22.615 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:29:22.615 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:29:22.616 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:29:22.616 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:29:22.616 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:29:27.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:29:27.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-15T20:29:27.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-15T20:29:27.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:29:27.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-15T20:29:27.934 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:29:27.935 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:29:27.935 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:29:27.935 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:29:27.935 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:29:27.943 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:29:27.944 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:29:27.944 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:29:27.945 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:29:27.946 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:29:27.946 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:29:27.946 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:29:27.947 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:29:27.947 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:29:27.947 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:29:27.948 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:29:27.948 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:29:27.949 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:29:27.950 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-15T20:29:27.972 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:29:27.972 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:29:27.973 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:29:27.973 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27419 2024-09-15T20:29:27.974 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:29:27.974 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:29:27.974 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-15T20:29:27.974 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-15T20:29:27.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-15T20:29:27.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-15T20:29:27.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-15T20:29:27.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:29:27.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-15T20:29:27.975 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:29:27.975 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:29:27.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:29:27.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:29:27.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:29:27.976 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:29:27.976 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:29:27.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:29:27.978 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:29:27.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:29:27.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:29:27.979 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:29:27.979 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:29:27.979 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:29:27.980 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:29:27.980 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:29:27.981 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:29:27.982 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:29:27.982 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-15T20:29:27.983 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:29:27.983 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:29:27.983 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:29:27.983 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27419 2024-09-15T20:29:27.984 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:29:27.984 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:29:27.984 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-15T20:29:27.985 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:29:27.986 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:29:27.986 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:29:27.986 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27419 2024-09-15T20:29:27.987 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:29:27.987 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T20:29:27.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:29:27.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-15T20:29:27.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2024-09-15T20:29:27.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2024-09-15T20:29:27.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2024-09-15T20:29:27.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2024-09-15T20:29:27.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2024-09-15T20:29:27.988 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2024-09-15T20:29:27.988 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2024-09-15T20:29:27.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2024-09-15T20:29:27.990 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:29:27.990 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:29:27.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2024-09-15T20:29:27.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-15T20:29:27.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T20:29:27.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T20:29:27.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T20:29:27.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-15T20:29:27.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-15T20:29:28.460 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T20:29:28.460 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:29:28.461 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:29:28.461 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:29:28.461 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:29:28.461 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:29:28.461 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:29:28.462 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T20:29:28.671 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T20:29:28.672 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T20:29:28.672 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:29:28.672 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:29:28.672 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T20:29:28.673 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T20:29:28.674 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:29:28.674 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:29:28.674 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:29:28.674 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:29:28.674 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:29:28.674 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:29:28.674 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-mon.a.asok 2024-09-15T20:29:28.674 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:29:28.674 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27419/ceph-mon.a.asok config get fsid 2024-09-15T20:29:28.788 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T20:29:28.789 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:29:28.789 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:29:28.789 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T20:29:28.789 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T20:29:28.790 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:29:28.790 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:29:28.790 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:29:28.791 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:29:28.791 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:29:28.791 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:29:28.791 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-mon.a.asok 2024-09-15T20:29:28.791 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:29:28.791 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27419/ceph-mon.a.asok config get mon_host 2024-09-15T20:29:28.906 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2024-09-15T20:29:28.906 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-15T20:29:28.906 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T20:29:28.907 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T20:29:28.907 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T20:29:28.907 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-15T20:29:28.907 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T20:29:29.817 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T20:29:29.817 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:29:29.817 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:29:29.818 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:29:29.818 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:29:29.818 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:29:29.818 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:29:29.819 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:29:29.820 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:29:29.858 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-15T20:29:29.858 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2024-09-15T20:29:29.858 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-15T20:29:29.858 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:29:29.858 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T20:29:29.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:29:29.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-15T20:29:29.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:29:29.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:29:29.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:29:29.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:29:29.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-15T20:29:29.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-15T20:29:29.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:29:29.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:29:29.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:29:29.859 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:29:29.860 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:29:29.860 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:29:29.860 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:29:29.860 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:29:29.860 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:29:29.860 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:29:29.860 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:29:29.860 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:29:29.860 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:29:29.860 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:29:29.860 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:29:29.860 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:29:29.860 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:29:29.860 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:29:29.861 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:29:29.861 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:29:29.861 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:29:29.861 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:29:29.861 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-15T20:29:29.861 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:29:29.861 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=457b5e59-ab0b-4ff7-8ba7-80b5ae6b10f2 2024-09-15T20:29:29.862 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 457b5e59-ab0b-4ff7-8ba7-80b5ae6b10f2' 2024-09-15T20:29:29.862 INFO:tasks.workunit.client.0.smithi139.stdout:add osd0 457b5e59-ab0b-4ff7-8ba7-80b5ae6b10f2 2024-09-15T20:29:29.862 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:29:29.883 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCpQ+dm4fmdNBAA7CFs6NzdyG7s0Db52Zd4vw== 2024-09-15T20:29:29.883 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCpQ+dm4fmdNBAA7CFs6NzdyG7s0Db52Zd4vw=="}' 2024-09-15T20:29:29.883 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 457b5e59-ab0b-4ff7-8ba7-80b5ae6b10f2 -i td/divergent-priors/0/new.json 2024-09-15T20:29:30.183 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:29:30.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-15T20:29:30.199 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCpQ+dm4fmdNBAA7CFs6NzdyG7s0Db52Zd4vw== --osd-uuid 457b5e59-ab0b-4ff7-8ba7-80b5ae6b10f2 2024-09-15T20:29:30.247 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:29:30.242+0000 7f5a2e9587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:29:30.250 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:29:30.246+0000 7f5a2e9587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:29:30.251 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:29:30.246+0000 7f5a2e9587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:29:30.251 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:29:30.246+0000 7f5a2e9587c0 -1 bdev(0x556a81033c00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:29:30.252 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:29:30.246+0000 7f5a2e9587c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-15T20:29:36.228 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-15T20:29:36.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:29:36.230 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd0 key to auth repository 2024-09-15T20:29:36.230 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T20:29:36.230 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:29:36.810 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T20:29:36.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T20:29:36.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:29:36.811 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:29:36.811 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:29:36.811 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:29:36.843 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:29:36.842+0000 7f1b7f0577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:29:36.848 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:29:36.846+0000 7f1b7f0577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:29:36.849 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:29:36.846+0000 7f1b7f0577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:29:37.226 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T20:29:37.226 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:29:37.227 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:29:37.227 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:29:37.227 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:29:37.227 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:29:37.227 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:29:37.227 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:29:37.228 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:29:37.228 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:29:37.703 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:29:38.357 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:29:38.354+0000 7f1b7f0577c0 -1 Falling back to public interface 2024-09-15T20:29:38.705 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:29:38.705 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:29:38.706 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:29:38.706 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:29:38.706 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:29:38.706 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:29:39.132 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:29:40.134 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:29:40.134 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:29:40.134 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:29:40.135 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:29:40.135 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:29:40.135 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:29:40.572 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:29:41.131 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:29:41.130+0000 7f1b7f0577c0 -1 osd.0 0 log_to_monitors true 2024-09-15T20:29:41.574 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:29:41.574 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:29:41.574 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:29:41.575 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:29:41.575 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:29:41.575 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:29:42.172 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:29:43.174 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:29:43.174 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:29:43.174 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:29:43.175 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:29:43.175 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:29:43.175 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:29:43.631 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:29:44.633 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:29:44.633 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:29:44.633 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:29:44.634 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:29:44.634 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:29:44.634 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:29:45.079 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:29:46.081 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:29:46.081 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:29:46.081 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:29:46.082 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:29:46.082 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:29:46.082 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:29:46.520 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:29:47.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:29:47.522 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:29:47.522 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:29:47.522 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:29:47.522 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:29:47.523 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:29:47.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:29:48.991 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:29:48.992 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:29:48.992 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:29:48.992 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:29:48.992 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:29:48.993 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:29:49.457 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:29:50.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:29:50.459 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:29:50.459 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:29:50.459 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:29:50.459 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:29:50.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:29:50.987 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:29:51.989 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:29:51.989 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:29:51.989 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:29:51.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:29:51.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:29:51.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:29:52.441 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:29:53.443 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:29:53.443 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:29:53.443 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:29:53.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:29:53.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:29:53.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:29:54.324 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:29:55.326 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:29:55.326 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:29:55.326 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:29:55.327 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:29:55.327 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:29:55.327 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:29:56.189 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:29:57.191 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:29:57.191 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:29:57.191 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:29:57.192 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:29:57.192 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:29:57.192 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:29:58.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:29:59.222 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:29:59.223 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:29:59.223 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:29:59.223 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:29:59.223 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:29:59.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:30:00.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:01.297 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:01.297 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:01.297 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:30:01.298 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:30:01.298 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:01.298 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:30:02.238 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:03.240 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:03.240 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:03.241 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:30:03.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:30:03.242 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:03.242 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:30:04.569 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:05.571 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:05.571 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:05.571 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:30:05.572 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:30:05.572 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:05.573 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:30:06.057 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1007282787,v1:127.0.0.1:6803/1007282787] [v2:127.0.0.1:6804/1007282787,v1:127.0.0.1:6805/1007282787] exists,up 457b5e59-ab0b-4ff7-8ba7-80b5ae6b10f2 2024-09-15T20:30:06.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:30:06.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:30:06.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:30:06.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-15T20:30:06.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2024-09-15T20:30:06.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-15T20:30:06.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:30:06.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T20:30:06.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:30:06.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-15T20:30:06.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:30:06.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:30:06.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:30:06.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:30:06.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T20:30:06.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T20:30:06.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:30:06.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:30:06.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:30:06.061 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:30:06.061 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:30:06.061 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:30:06.061 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:30:06.061 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:30:06.062 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:30:06.062 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:30:06.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:30:06.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:30:06.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:30:06.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:30:06.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:30:06.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:30:06.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:30:06.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:30:06.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:30:06.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:30:06.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:30:06.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:30:06.064 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-15T20:30:06.064 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:30:06.064 INFO:tasks.workunit.client.0.smithi139.stdout:add osd1 037d24c1-96ec-4453-b5af-a66b6de4e720 2024-09-15T20:30:06.064 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=037d24c1-96ec-4453-b5af-a66b6de4e720 2024-09-15T20:30:06.065 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 037d24c1-96ec-4453-b5af-a66b6de4e720' 2024-09-15T20:30:06.065 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:30:06.087 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDOQ+dmp2ArBRAAzYLHaN3onj/OD5XdZ2VBfQ== 2024-09-15T20:30:06.087 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDOQ+dmp2ArBRAAzYLHaN3onj/OD5XdZ2VBfQ=="}' 2024-09-15T20:30:06.087 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 037d24c1-96ec-4453-b5af-a66b6de4e720 -i td/divergent-priors/1/new.json 2024-09-15T20:30:06.656 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:30:06.673 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-15T20:30:06.674 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDOQ+dmp2ArBRAAzYLHaN3onj/OD5XdZ2VBfQ== --osd-uuid 037d24c1-96ec-4453-b5af-a66b6de4e720 2024-09-15T20:30:06.708 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:06.706+0000 7fe5be1b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:06.714 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:06.714+0000 7fe5be1b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:06.715 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:06.714+0000 7fe5be1b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:06.716 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:06.714+0000 7fe5be1b47c0 -1 bdev(0x55997d578800 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:30:06.716 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:06.714+0000 7fe5be1b47c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-15T20:30:12.755 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-15T20:30:12.755 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:30:12.756 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd1 key to auth repository 2024-09-15T20:30:12.757 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T20:30:12.757 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:30:13.335 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.1 2024-09-15T20:30:13.336 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T20:30:13.336 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:30:13.336 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:30:13.336 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:30:13.337 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:30:13.376 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:13.374+0000 7f4ed4c3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:13.394 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:13.394+0000 7f4ed4c3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:13.395 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:13.394+0000 7f4ed4c3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:13.804 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T20:30:13.805 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:30:13.805 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:30:13.805 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:30:13.805 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:30:13.805 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:30:13.806 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:13.806 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:30:13.806 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:13.806 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:30:14.264 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:14.444 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:14.441+0000 7f4ed4c3a7c0 -1 Falling back to public interface 2024-09-15T20:30:15.265 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:15.266 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:15.266 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:30:15.266 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:30:15.266 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:15.267 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:30:15.749 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:16.751 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:16.751 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:16.751 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:30:16.752 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:30:16.752 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:16.752 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:30:17.301 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:17.297+0000 7f4ed4c3a7c0 -1 osd.1 0 log_to_monitors true 2024-09-15T20:30:17.324 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:18.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:18.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:18.325 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:30:18.326 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:30:18.326 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:18.326 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:30:18.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:19.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:19.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:19.787 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:30:19.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:30:19.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:19.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:30:20.447 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:21.449 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:21.449 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:21.449 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:30:21.450 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:30:21.450 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:21.450 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:30:22.131 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:23.133 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:23.133 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:23.134 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:30:23.134 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:30:23.134 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:23.134 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:30:23.615 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:24.617 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:24.617 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:24.617 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:30:24.618 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:30:24.618 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:24.618 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:30:25.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:26.097 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:26.097 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:26.098 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:30:26.098 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:30:26.098 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:26.098 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:30:26.546 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:27.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:27.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:27.549 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:30:27.549 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:30:27.549 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:27.549 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:30:28.058 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:29.059 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:29.059 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:29.060 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:30:29.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:30:29.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:29.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:30:29.746 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:30.748 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:30:30.748 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:30.748 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:30.748 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:30:30.748 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:30:30.749 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:31.907 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:32.909 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:32.909 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:32.909 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:30:32.910 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:30:32.910 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:32.910 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:30:33.715 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:34.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:34.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:34.717 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:30:34.717 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:30:34.717 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:34.717 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:30:35.646 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:36.648 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:36.648 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:36.648 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:30:36.649 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:30:36.649 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:36.649 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:30:37.432 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:38.434 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:38.434 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:38.434 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:30:38.435 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:30:38.435 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:38.435 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:30:39.703 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:40.705 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:40.705 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:40.705 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:30:40.706 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:30:40.706 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:40.706 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:30:41.273 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4110363107,v1:127.0.0.1:6811/4110363107] [v2:127.0.0.1:6812/4110363107,v1:127.0.0.1:6813/4110363107] exists,up 037d24c1-96ec-4453-b5af-a66b6de4e720 2024-09-15T20:30:41.276 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:30:41.276 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:30:41.276 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:30:41.276 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-15T20:30:41.276 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2024-09-15T20:30:41.276 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-15T20:30:41.276 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:30:41.276 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T20:30:41.277 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:30:41.277 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-15T20:30:41.277 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:30:41.277 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:30:41.277 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:30:41.277 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:30:41.277 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-15T20:30:41.277 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-15T20:30:41.277 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:30:41.277 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:30:41.277 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:30:41.277 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:30:41.278 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:30:41.278 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:30:41.278 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:30:41.278 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:30:41.278 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:30:41.278 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:30:41.278 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:30:41.278 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:30:41.278 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:30:41.278 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:30:41.278 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:30:41.278 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:30:41.278 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:30:41.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:30:41.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:30:41.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:30:41.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:30:41.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:30:41.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-15T20:30:41.291 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:30:41.292 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=24e3e077-bb9a-4ae3-9be6-20ce6b775ae4 2024-09-15T20:30:41.292 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 24e3e077-bb9a-4ae3-9be6-20ce6b775ae4' 2024-09-15T20:30:41.292 INFO:tasks.workunit.client.0.smithi139.stdout:add osd2 24e3e077-bb9a-4ae3-9be6-20ce6b775ae4 2024-09-15T20:30:41.293 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:30:41.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDxQ+dm4WLUEhAASI0kGJ4JqxbTzix7emRY7Q== 2024-09-15T20:30:41.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDxQ+dm4WLUEhAASI0kGJ4JqxbTzix7emRY7Q=="}' 2024-09-15T20:30:41.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 24e3e077-bb9a-4ae3-9be6-20ce6b775ae4 -i td/divergent-priors/2/new.json 2024-09-15T20:30:41.833 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:30:41.849 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-15T20:30:41.851 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDxQ+dm4WLUEhAASI0kGJ4JqxbTzix7emRY7Q== --osd-uuid 24e3e077-bb9a-4ae3-9be6-20ce6b775ae4 2024-09-15T20:30:41.886 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:41.885+0000 7f13cf79d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:41.888 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:41.885+0000 7f13cf79d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:41.889 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:41.885+0000 7f13cf79d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:41.889 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:41.889+0000 7f13cf79d7c0 -1 bdev(0x561fb0225c00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-15T20:30:41.889 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:41.889+0000 7f13cf79d7c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-15T20:30:48.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-15T20:30:48.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:30:48.122 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd2 key to auth repository 2024-09-15T20:30:48.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T20:30:48.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:30:48.713 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.2 2024-09-15T20:30:48.714 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T20:30:48.714 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:30:48.714 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:30:48.714 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:30:48.714 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:30:48.755 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:48.753+0000 7fab0b9d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:48.786 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:48.785+0000 7fab0b9d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:48.788 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:48.785+0000 7fab0b9d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:30:49.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T20:30:49.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:30:49.174 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:30:49.174 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:30:49.174 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:30:49.174 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:49.174 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:30:49.174 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:30:49.175 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:49.175 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:30:49.649 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:50.280 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:50.277+0000 7fab0b9d37c0 -1 Falling back to public interface 2024-09-15T20:30:50.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:50.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:50.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:30:50.651 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:30:50.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:50.652 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:30:51.167 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:52.168 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:52.168 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:52.169 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:30:52.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:30:52.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:52.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:30:52.603 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:53.187 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:30:53.185+0000 7fab0b9d37c0 -1 osd.2 0 log_to_monitors true 2024-09-15T20:30:53.605 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:53.605 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:53.605 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:30:53.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:30:53.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:53.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:30:54.323 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:55.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:55.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:55.326 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:30:55.326 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:30:55.326 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:55.326 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:30:55.793 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:56.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:56.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:56.795 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:30:56.796 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:30:56.796 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:56.796 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:30:57.461 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:30:58.463 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:30:58.463 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:30:58.464 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:30:58.464 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:30:58.464 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:30:58.464 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:30:59.105 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:00.106 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:00.107 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:00.107 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:31:00.107 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:31:00.107 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:00.108 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:31:00.595 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:01.597 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:01.597 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:01.597 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:31:01.598 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:31:01.598 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:01.598 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:31:02.107 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:03.109 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:03.109 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:03.109 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:31:03.110 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:31:03.110 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:03.110 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:31:03.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:04.718 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:04.718 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:04.718 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:31:04.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:31:04.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:04.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:31:05.481 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:06.482 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:06.483 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:06.483 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:31:06.483 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:31:06.483 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:06.483 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:31:07.626 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:08.627 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:08.628 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:08.628 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:31:08.628 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:31:08.629 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:08.629 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:31:09.944 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:10.946 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:10.946 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:10.946 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:31:10.946 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:31:10.947 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:10.947 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:31:11.960 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:12.962 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:12.963 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:12.963 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:31:12.963 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:31:12.963 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:12.963 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:31:14.070 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:15.072 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:15.072 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:15.072 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:31:15.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:31:15.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:15.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:31:15.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:31:16.526 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:31:16.526 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:31:16.526 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:31:16.527 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:31:16.527 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:31:16.527 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:31:17.022 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2687036097,v1:127.0.0.1:6819/2687036097] [v2:127.0.0.1:6820/2687036097,v1:127.0.0.1:6821/2687036097] exists,up 24e3e077-bb9a-4ae3-9be6-20ce6b775ae4 2024-09-15T20:31:17.022 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:31:17.022 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:31:17.022 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:31:17.023 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2024-09-15T20:31:17.813 INFO:tasks.workunit.client.0.smithi139.stderr:noout is set 2024-09-15T20:31:17.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2024-09-15T20:31:18.650 INFO:tasks.workunit.client.0.smithi139.stderr:noin is set 2024-09-15T20:31:18.669 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2024-09-15T20:31:19.587 INFO:tasks.workunit.client.0.smithi139.stderr:nodown is set 2024-09-15T20:31:19.607 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2024-09-15T20:31:19.608 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-15T20:31:20.378 INFO:tasks.workunit.client.0.smithi139.stderr:pool 'test' already exists 2024-09-15T20:31:20.399 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:31:21.401 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2024-09-15T20:31:22.206 INFO:tasks.workunit.client.0.smithi139.stderr:set pool 1 size to 3 2024-09-15T20:31:22.227 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2024-09-15T20:31:22.901 INFO:tasks.workunit.client.0.smithi139.stderr:set pool 1 min_size to 2 2024-09-15T20:31:22.923 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2024-09-15T20:31:22.923 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:31:22.924 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:31:23.449 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:31:23.450 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:31:23.450 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:31:23.450 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:31:23.450 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:31:23.450 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:31:23.616 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-15T20:31:23.616 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-15T20:31:23.616 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-15T20:31:23.616 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:31:23.616 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:31:23.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672974 2024-09-15T20:31:23.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672974 2024-09-15T20:31:23.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672974' 2024-09-15T20:31:23.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:31:23.786 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:31:23.937 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-15T20:31:23.938 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-15T20:31:23.938 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672974 2-64424509448' 2024-09-15T20:31:23.938 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:31:23.938 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-15T20:31:23.938 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:31:23.939 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:31:23.939 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-15T20:31:23.939 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:31:23.940 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-15T20:31:23.940 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-15T20:31:23.941 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 21474836501 2024-09-15T20:31:23.941 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:31:24.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836501 2024-09-15T20:31:24.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:31:25.419 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:31:25.419 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:31:25.851 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-15T20:31:25.851 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:31:25.851 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672974 2024-09-15T20:31:25.851 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:31:25.853 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:31:25.854 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672974 2024-09-15T20:31:25.854 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:31:25.855 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672974 2024-09-15T20:31:25.855 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.1 seq 42949672974 2024-09-15T20:31:25.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-15T20:31:25.856 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:31:26.282 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672974 -lt 42949672974 2024-09-15T20:31:26.282 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:31:26.282 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:31:26.283 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-15T20:31:26.284 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:31:26.285 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-15T20:31:26.285 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:31:26.285 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-15T20:31:26.286 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 64424509448 2024-09-15T20:31:26.286 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-15T20:31:26.286 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:31:26.726 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-15T20:31:26.726 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2024-09-15T20:31:26.726 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:31:26.726 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:31:26.726 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:31:26.726 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:31:26.727 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:31:26.727 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:31:26.727 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:31:26.727 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:31:26.728 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:31:26.833 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:31:26.833 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:31:26.833 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:31:26.833 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:31:26.833 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:31:26.833 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:31:27.298 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:31:27.298 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:31:27.298 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:31:27.298 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:31:27.298 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:31:27.298 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:31:27.474 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-15T20:31:27.474 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-15T20:31:27.475 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-15T20:31:27.475 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:31:27.475 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:31:27.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-15T20:31:27.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-15T20:31:27.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672977' 2024-09-15T20:31:27.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:31:27.622 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:31:27.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-15T20:31:27.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-15T20:31:27.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672977 2-64424509451' 2024-09-15T20:31:27.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:31:27.785 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-15T20:31:27.785 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:31:27.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:31:27.786 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-15T20:31:27.786 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:31:27.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-15T20:31:27.787 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 21474836504 2024-09-15T20:31:27.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-15T20:31:27.788 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:31:28.267 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836504 2024-09-15T20:31:28.267 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:31:29.269 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:31:29.269 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:31:29.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-15T20:31:29.760 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:31:29.760 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-15T20:31:29.761 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:31:29.762 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:31:29.762 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-15T20:31:29.762 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:31:29.763 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-15T20:31:29.764 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.1 seq 42949672977 2024-09-15T20:31:29.764 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-15T20:31:29.764 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:31:30.211 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-15T20:31:30.212 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:31:30.212 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-15T20:31:30.212 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:31:30.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:31:30.214 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-15T20:31:30.214 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:31:30.215 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-15T20:31:30.216 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 64424509451 2024-09-15T20:31:30.216 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-15T20:31:30.216 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:31:30.669 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-15T20:31:30.669 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:31:30.669 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:31:30.669 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:31:31.223 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:31:31.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:31:31.225 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:31:31.225 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:31:31.225 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:31:31.225 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:31:31.225 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:31:31.225 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:31:31.641 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:31:31.642 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:31:31.643 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:31:31.643 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:31:32.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:31:32.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:31:32.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:31:32.215 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2024-09-15T20:31:32.215 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2024-09-15T20:31:32.634 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:31:32.650 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2024-09-15T20:31:32.651 INFO:tasks.workunit.client.0.smithi139.stdout:primary and soon to be divergent is 1 2024-09-15T20:31:32.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2024-09-15T20:31:32.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2024-09-15T20:31:33.046 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:31:33.046 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T20:31:21.580660+0000 0'0 29:20 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:31:19.998451+0000 0'0 2024-09-15T20:31:19.998451+0000 0 0 periodic scrub scheduled @ 2024-09-16T21:35:45.476802+0000 0 0 2024-09-15T20:31:33.046 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:31:33.046 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:31:33.046 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:31:33.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2024-09-15T20:31:33.062 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-15T20:31:33.062 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2024-09-15T20:31:33.062 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2024-09-15T20:31:33.062 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-15T20:31:33.062 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2024-09-15T20:31:33.062 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2024-09-15T20:31:33.062 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-15T20:31:33.062 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2024-09-15T20:31:33.062 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2024-09-15T20:31:33.062 INFO:tasks.workunit.client.0.smithi139.stdout:writing initial objects 2024-09-15T20:31:33.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2024-09-15T20:31:33.063 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2024-09-15T20:31:33.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:33.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2024-09-15T20:31:33.216 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:33.216 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2024-09-15T20:31:33.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:33.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2024-09-15T20:31:33.518 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:33.518 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2024-09-15T20:31:33.670 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:33.670 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2024-09-15T20:31:33.820 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:33.820 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2024-09-15T20:31:33.979 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:33.979 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2024-09-15T20:31:34.164 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:34.164 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2024-09-15T20:31:34.323 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:34.323 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2024-09-15T20:31:34.473 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:34.473 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2024-09-15T20:31:34.633 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:34.633 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2024-09-15T20:31:34.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:34.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2024-09-15T20:31:34.935 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:34.935 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2024-09-15T20:31:35.119 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:35.119 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2024-09-15T20:31:35.269 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:35.270 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2024-09-15T20:31:35.438 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:35.438 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2024-09-15T20:31:35.587 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:35.588 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2024-09-15T20:31:35.739 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:35.739 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2024-09-15T20:31:35.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:35.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2024-09-15T20:31:36.033 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:36.033 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2024-09-15T20:31:36.183 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:36.183 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2024-09-15T20:31:36.342 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:36.343 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2024-09-15T20:31:36.501 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:36.501 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2024-09-15T20:31:36.669 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:36.669 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2024-09-15T20:31:36.819 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:36.819 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2024-09-15T20:31:37.003 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:37.003 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2024-09-15T20:31:37.180 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:37.181 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2024-09-15T20:31:37.339 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:37.339 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2024-09-15T20:31:37.481 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:37.481 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2024-09-15T20:31:37.623 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:37.623 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2024-09-15T20:31:37.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:37.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2024-09-15T20:31:37.968 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:37.968 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2024-09-15T20:31:38.127 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:38.127 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2024-09-15T20:31:38.285 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:38.285 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2024-09-15T20:31:38.461 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:38.461 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2024-09-15T20:31:38.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:38.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2024-09-15T20:31:38.771 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:38.771 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2024-09-15T20:31:38.914 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:38.914 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2024-09-15T20:31:39.065 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:39.065 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2024-09-15T20:31:39.207 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:39.207 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2024-09-15T20:31:39.358 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:39.358 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2024-09-15T20:31:39.517 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:39.517 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2024-09-15T20:31:39.677 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:39.677 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2024-09-15T20:31:39.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:39.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2024-09-15T20:31:39.994 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:39.995 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2024-09-15T20:31:40.146 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:40.146 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2024-09-15T20:31:40.288 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:40.288 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2024-09-15T20:31:40.447 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:40.447 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2024-09-15T20:31:40.615 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:40.615 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2024-09-15T20:31:40.757 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:40.757 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2024-09-15T20:31:40.916 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:40.916 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2024-09-15T20:31:41.076 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:41.076 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2024-09-15T20:31:41.235 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:41.235 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2024-09-15T20:31:41.394 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:41.394 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2024-09-15T20:31:41.553 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:41.553 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2024-09-15T20:31:41.704 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:41.704 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2024-09-15T20:31:41.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:41.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2024-09-15T20:31:42.039 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:42.039 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2024-09-15T20:31:42.181 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:42.181 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2024-09-15T20:31:42.332 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:42.332 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2024-09-15T20:31:42.492 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:42.492 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2024-09-15T20:31:42.658 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:42.659 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2024-09-15T20:31:42.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:42.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2024-09-15T20:31:42.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:42.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2024-09-15T20:31:43.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:43.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2024-09-15T20:31:43.322 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:43.322 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2024-09-15T20:31:43.480 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:43.480 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2024-09-15T20:31:43.631 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:43.631 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2024-09-15T20:31:43.798 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:43.798 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2024-09-15T20:31:43.942 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:43.942 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2024-09-15T20:31:44.100 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:44.100 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2024-09-15T20:31:44.259 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:44.259 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2024-09-15T20:31:44.419 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:44.419 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2024-09-15T20:31:44.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:44.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2024-09-15T20:31:44.737 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:44.737 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2024-09-15T20:31:44.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:44.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2024-09-15T20:31:45.048 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:45.048 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2024-09-15T20:31:45.189 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:45.189 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2024-09-15T20:31:45.348 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:45.348 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2024-09-15T20:31:45.516 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:45.516 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2024-09-15T20:31:45.684 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:45.684 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2024-09-15T20:31:45.843 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:45.843 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2024-09-15T20:31:46.002 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:46.002 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2024-09-15T20:31:46.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:46.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2024-09-15T20:31:46.337 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:46.337 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2024-09-15T20:31:46.487 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:46.487 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2024-09-15T20:31:46.663 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:46.663 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2024-09-15T20:31:46.814 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:46.814 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2024-09-15T20:31:46.981 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:46.982 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2024-09-15T20:31:47.183 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:47.184 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2024-09-15T20:31:47.342 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:47.342 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2024-09-15T20:31:47.509 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:47.509 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2024-09-15T20:31:47.669 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:47.669 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2024-09-15T20:31:47.829 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:47.829 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2024-09-15T20:31:47.979 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:47.980 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2024-09-15T20:31:48.130 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:48.130 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2024-09-15T20:31:48.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:48.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2024-09-15T20:31:48.423 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:48.423 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2024-09-15T20:31:48.565 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:48.565 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2024-09-15T20:31:48.734 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T20:31:48.734 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2024-09-15T20:31:48.892 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T20:31:48.892 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2024-09-15T20:31:48.892 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:31:48.893 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:31:48.893 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:31:48.893 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T20:31:48.894 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:31:48.894 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:31:48.895 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:31:48.895 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:31:48.895 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:31:48.970 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T20:31:48.970 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:31:48.970 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:31:48.971 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:31:48.971 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:31:48.971 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:31:49.470 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:31:49.470 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:31:49.470 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:31:49.470 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:31:49.470 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:31:49.470 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:31:49.639 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836510 2024-09-15T20:31:49.639 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836510 2024-09-15T20:31:49.639 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510' 2024-09-15T20:31:49.639 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:31:49.639 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:31:49.806 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672983 2024-09-15T20:31:49.806 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672983 2024-09-15T20:31:49.807 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672983' 2024-09-15T20:31:49.807 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:31:49.807 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:31:49.974 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509457 2024-09-15T20:31:49.974 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509457 2024-09-15T20:31:49.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672983 2-64424509457' 2024-09-15T20:31:49.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:31:49.975 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836510 2024-09-15T20:31:49.975 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:31:49.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:31:49.978 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836510 2024-09-15T20:31:49.978 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:31:49.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836510 2024-09-15T20:31:49.979 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 21474836510 2024-09-15T20:31:49.979 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-15T20:31:49.979 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:31:50.434 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836510 2024-09-15T20:31:50.434 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:31:51.436 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:31:51.436 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:31:51.921 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836510 -lt 21474836510 2024-09-15T20:31:51.921 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:31:51.922 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672983 2024-09-15T20:31:51.922 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:31:51.924 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:31:51.925 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672983 2024-09-15T20:31:51.925 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:31:51.926 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672983 2024-09-15T20:31:51.927 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.1 seq 42949672983 2024-09-15T20:31:51.927 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-15T20:31:51.927 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:31:52.373 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-15T20:31:52.374 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:31:52.374 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509457 2024-09-15T20:31:52.374 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:31:52.376 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:31:52.377 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509457 2024-09-15T20:31:52.377 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:31:52.379 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509457 2024-09-15T20:31:52.379 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 64424509457 2024-09-15T20:31:52.379 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-15T20:31:52.379 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:31:52.806 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509458 -lt 64424509457 2024-09-15T20:31:52.809 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:31:52.810 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:31:52.810 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:31:53.372 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:31:53.373 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:31:53.373 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:31:53.373 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:31:53.374 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:31:53.374 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:31:53.374 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:31:53.374 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:31:53.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:31:53.795 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:31:53.796 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:31:53.796 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:31:54.341 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:31:54.341 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:31:54.341 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:31:54.342 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2024-09-15T20:31:54.342 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-15T20:31:54.342 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-15T20:31:54.342 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-15T20:31:54.342 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:31:54.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2024-09-15T20:31:54.787 INFO:tasks.workunit.client.0.smithi139.stdout:blackholing osds 0 2 2024-09-15T20:31:54.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2024-09-15T20:31:54.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2024-09-15T20:31:55.203 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:31:55.204 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-09-15T20:31:21.580660+0000 29'100 29:120 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:31:19.998451+0000 0'0 2024-09-15T20:31:19.998451+0000 0 0 periodic scrub scheduled @ 2024-09-16T21:35:45.476802+0000 0 0 2024-09-15T20:31:55.204 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:31:55.204 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:31:55.204 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:31:55.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-09-15T20:31:55.221 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2024-09-15T20:31:55.221 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T20:31:55.221 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T20:31:55.222 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:31:55.222 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:31:55.222 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:31:55.222 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.0.asok 2024-09-15T20:31:55.223 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-09-15T20:31:55.223 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-15T20:31:55.326 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:31:55.326 INFO:tasks.workunit.client.0.smithi139.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-15T20:31:55.326 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:31:55.337 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-09-15T20:31:55.338 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2024-09-15T20:31:55.338 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-15T20:31:55.338 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-15T20:31:55.339 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:31:55.339 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:31:55.339 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:31:55.339 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.2.asok 2024-09-15T20:31:55.340 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-09-15T20:31:55.340 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-15T20:31:55.435 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:31:55.435 INFO:tasks.workunit.client.0.smithi139.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-15T20:31:55.435 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:31:55.446 INFO:tasks.workunit.client.0.smithi139.stdout:create new divergent objects 2024-09-15T20:31:55.446 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2024-09-15T20:31:55.446 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2024-09-15T20:31:55.448 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-15T20:31:55.448 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2024-09-15T20:31:55.449 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-15T20:31:55.449 INFO:tasks.workunit.client.0.smithi139.stdout:writing divergent objects 2024-09-15T20:31:55.449 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2024-09-15T20:31:55.449 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2024-09-15T20:31:55.450 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2024-09-15T20:31:55.451 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-15T20:31:55.452 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-15T20:31:55.452 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2024-09-15T20:31:55.452 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2024-09-15T20:31:55.452 INFO:tasks.workunit.client.0.smithi139.stdout:remove divergent objects 2024-09-15T20:31:55.453 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2024-09-15T20:31:55.453 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2024-09-15T20:31:55.454 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-15T20:31:55.455 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2024-09-15T20:31:55.456 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2024-09-15T20:31:55.456 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-15T20:31:55.457 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2024-09-15T20:31:55.457 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2024-09-15T20:31:55.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2024-09-15T20:31:55.459 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2024-09-15T20:31:55.459 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2024-09-15T20:32:05.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2024-09-15T20:32:05.466 INFO:tasks.workunit.client.0.smithi139.stderr:rados: no process found 2024-09-15T20:32:05.466 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2024-09-15T20:32:05.467 INFO:tasks.workunit.client.0.smithi139.stdout:killing all the osds 2024-09-15T20:32:05.467 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2024-09-15T20:32:05.888 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:32:05.889 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 active+clean 2024-09-15T20:31:21.580660+0000 29'106 29:126 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:31:19.998451+0000 0'0 2024-09-15T20:31:19.998451+0000 0 0 periodic scrub scheduled @ 2024-09-16T21:35:45.476802+0000 0 0 2024-09-15T20:32:05.889 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:32:05.889 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:32:05.889 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:32:05.905 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2024-09-15T20:32:05.906 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:32:05.906 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:32:05.907 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:32:05.907 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:32:05.907 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:32:06.016 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:32:06.016 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-15T20:32:06.016 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2024-09-15T20:32:06.531 INFO:tasks.workunit.client.0.smithi139.stderr:osd.0 is already down. 2024-09-15T20:32:06.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-15T20:32:06.548 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2024-09-15T20:32:07.189 INFO:tasks.workunit.client.0.smithi139.stderr:osd.1 is already down. 2024-09-15T20:32:07.207 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-15T20:32:07.207 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2024-09-15T20:32:07.778 INFO:tasks.workunit.client.0.smithi139.stderr:osd.2 is already down. 2024-09-15T20:32:07.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-09-15T20:32:07.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2024-09-15T20:32:08.345 INFO:tasks.workunit.client.0.smithi139.stderr:osd.0 is already out. 2024-09-15T20:32:08.362 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-09-15T20:32:08.363 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2024-09-15T20:32:09.002 INFO:tasks.workunit.client.0.smithi139.stderr:osd.2 is already out. 2024-09-15T20:32:09.018 INFO:tasks.workunit.client.0.smithi139.stdout:bringing up non_divergent 0 2 2024-09-15T20:32:09.019 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2024-09-15T20:32:09.019 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2024-09-15T20:32:09.440 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:32:09.440 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 stale+active+clean 2024-09-15T20:31:21.580660+0000 29'106 29:126 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:31:19.998451+0000 0'0 2024-09-15T20:31:19.998451+0000 0 0 periodic scrub scheduled @ 2024-09-16T21:35:45.476802+0000 0 0 2024-09-15T20:32:09.440 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:32:09.440 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:32:09.440 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:32:09.457 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-09-15T20:32:09.457 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2024-09-15T20:32:09.457 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-15T20:32:09.457 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:32:09.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:32:09.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:32:09.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-15T20:32:09.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:32:09.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:32:09.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:32:09.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:32:09.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-15T20:32:09.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-15T20:32:09.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:32:09.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:32:09.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:32:09.458 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:32:09.459 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:32:09.459 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:32:09.459 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:32:09.459 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:32:09.459 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:32:09.459 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:32:09.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:32:09.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:32:09.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:32:09.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:32:09.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:32:09.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:32:09.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:32:09.461 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:32:09.461 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:32:09.461 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:32:09.461 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-15T20:32:09.461 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T20:32:09.462 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:32:09.462 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:32:09.462 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-15T20:32:09.462 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:32:09.462 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:32:09.462 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:32:09.462 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:32:09.508 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:32:09.504+0000 7f62f14fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:32:09.509 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:32:09.508+0000 7f62f14fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:32:09.510 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:32:09.508+0000 7f62f14fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:32:09.906 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:32:09.906 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:32:09.906 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:32:09.906 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:32:09.906 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:32:09.906 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:09.906 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:32:09.907 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:32:09.907 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:09.907 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:32:10.375 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:10.575 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:32:10.572+0000 7f62f14fe7c0 -1 Falling back to public interface 2024-09-15T20:32:11.377 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:11.377 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:11.377 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:32:11.378 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:32:11.378 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:11.378 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:32:11.877 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:12.878 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:32:12.879 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:12.879 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:12.879 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:32:12.879 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:12.880 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:32:13.327 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:13.832 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:32:13.828+0000 7f62f14fe7c0 -1 osd.0 29 log_to_monitors true 2024-09-15T20:32:14.329 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:14.329 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:14.329 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:32:14.330 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:32:14.330 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:14.330 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:32:14.781 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/1803718560,v1:127.0.0.1:6803/1803718560] [v2:127.0.0.1:6804/1803718560,v1:127.0.0.1:6805/1803718560] exists,up 457b5e59-ab0b-4ff7-8ba7-80b5ae6b10f2 2024-09-15T20:32:14.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:32:14.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:32:14.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:32:14.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-09-15T20:32:14.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2024-09-15T20:32:14.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-15T20:32:14.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:32:14.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T20:32:14.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:32:14.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-15T20:32:14.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:32:14.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:32:14.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:32:14.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:32:14.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-15T20:32:14.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-15T20:32:14.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:32:14.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:32:14.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:32:14.785 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:32:14.785 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:32:14.785 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:32:14.785 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:32:14.785 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:32:14.785 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:32:14.786 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:32:14.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:32:14.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:32:14.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:32:14.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:32:14.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:32:14.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:32:14.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:32:14.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:32:14.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:32:14.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:32:14.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-15T20:32:14.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T20:32:14.787 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.2 2024-09-15T20:32:14.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:32:14.788 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-15T20:32:14.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T20:32:14.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:32:14.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:32:14.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:32:14.818 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:32:14.816+0000 7f7731dae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:32:14.825 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:32:14.824+0000 7f7731dae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:32:14.827 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:32:14.824+0000 7f7731dae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:32:15.249 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T20:32:15.249 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:32:15.249 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:32:15.249 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:32:15.249 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:32:15.249 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:15.250 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:32:15.250 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:32:15.250 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:15.250 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:15.663 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:16.059 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:32:16.056+0000 7f7731dae7c0 -1 Falling back to public interface 2024-09-15T20:32:16.664 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:16.664 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:16.665 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:32:16.665 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:32:16.665 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:16.665 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:17.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:18.175 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:18.175 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:18.175 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:32:18.176 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:32:18.176 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:18.176 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:18.618 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:32:19.349 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:32:19.348+0000 7f7731dae7c0 -1 osd.2 29 log_to_monitors true 2024-09-15T20:32:19.620 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:32:19.620 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:32:19.621 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:32:19.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:32:19.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:32:19.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:32:20.179 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/3470466229,v1:127.0.0.1:6811/3470466229] [v2:127.0.0.1:6812/3470466229,v1:127.0.0.1:6813/3470466229] exists,up 24e3e077-bb9a-4ae3-9be6-20ce6b775ae4 2024-09-15T20:32:20.179 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:32:20.179 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:32:20.179 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:32:20.180 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-09-15T20:32:20.180 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2024-09-15T20:32:20.769 INFO:tasks.workunit.client.0.smithi139.stderr:osd.0 is already in. 2024-09-15T20:32:20.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-09-15T20:32:20.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2024-09-15T20:32:21.590 INFO:tasks.workunit.client.0.smithi139.stderr:osd.2 is already in. 2024-09-15T20:32:21.611 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T20:32:21.612 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2024-09-15T20:32:21.612 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:32:21.612 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:32:21.612 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:32:21.612 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T20:32:21.613 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:32:21.613 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:32:21.613 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:32:21.613 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:32:21.613 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:32:21.684 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T20:32:21.685 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:32:21.685 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:32:21.685 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:32:21.685 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:32:21.685 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:32:22.177 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:32:22.177 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:32:22.177 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:32:22.177 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:32:22.177 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:32:22.177 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:32:22.352 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822660 2024-09-15T20:32:22.353 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822660 2024-09-15T20:32:22.353 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660' 2024-09-15T20:32:22.353 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:32:22.353 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:32:22.477 INFO:tasks.workunit.client.0.smithi139.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-15T20:32:22.480 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-15T20:32:22.480 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-15T20:32:22.480 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-15T20:32:22.480 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:32:22.480 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:32:22.640 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757251 2024-09-15T20:32:22.640 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757251 2024-09-15T20:32:22.640 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-09-15T20:32:22.640 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:32:22.640 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-154618822660 2024-09-15T20:32:22.640 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:32:22.641 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:32:22.642 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-154618822660 2024-09-15T20:32:22.642 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:32:22.642 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822660 2024-09-15T20:32:22.643 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 154618822660 2024-09-15T20:32:22.643 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-09-15T20:32:22.643 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:32:23.086 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822660 -lt 154618822660 2024-09-15T20:32:23.086 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:32:23.087 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-163208757251 2024-09-15T20:32:23.088 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:32:23.089 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:32:23.090 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-163208757251 2024-09-15T20:32:23.090 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:32:23.091 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757251 2024-09-15T20:32:23.091 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 163208757251 2024-09-15T20:32:23.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-09-15T20:32:23.092 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:32:23.563 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 163208757251 2024-09-15T20:32:23.563 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:32:24.565 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:32:24.565 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:32:25.031 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757251 -lt 163208757251 2024-09-15T20:32:25.032 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:32:25.032 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:32:25.032 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:32:25.600 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:32:25.600 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:32:25.601 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:32:25.601 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:32:25.601 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:32:25.601 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:32:25.601 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:32:25.601 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:32:26.033 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:32:26.033 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:32:26.034 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:32:26.034 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:32:26.612 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:32:26.613 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T20:32:26.613 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:32:26.613 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T20:32:26.613 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:32:26.613 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:32:26.714 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:32:26.714 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:32:26.715 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:32:26.715 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:32:26.715 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:32:26.715 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:32:26.715 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:32:26.716 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:32:27.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:32:27.129 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:32:27.130 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:32:27.130 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:32:27.688 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:32:27.689 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:32:27.689 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:32:27.689 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:32:27.689 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:32:27.689 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:32:27.689 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:32:27.689 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:32:27.689 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:32:28.268 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:32:28.268 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:32:28.268 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-15T20:32:28.268 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:32:28.268 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:32:28.469 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:32:28.469 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:32:28.470 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:32:28.470 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:32:28.470 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:32:28.470 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:32:28.471 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:32:28.471 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:32:28.901 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:32:28.901 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:32:28.902 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:32:28.902 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:32:29.476 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:32:29.476 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:32:29.476 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:32:29.476 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:32:29.476 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:32:29.476 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:32:29.476 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:32:29.476 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:32:29.477 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:32:30.054 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:32:30.054 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:32:30.055 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-15T20:32:30.055 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:32:30.055 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T20:32:30.456 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:32:30.456 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:32:30.456 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:32:30.456 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:32:30.457 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:32:30.457 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:32:30.457 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:32:30.457 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:32:30.892 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:32:30.893 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:32:30.893 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:32:30.893 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:32:31.478 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:32:31.478 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:32:31.478 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:32:31.478 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:32:31.478 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:32:31.478 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:32:31.478 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:32:31.479 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:32:31.479 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:32:32.028 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:32:32.028 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:32:32.029 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-15T20:32:32.029 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:32:32.029 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-15T20:32:32.830 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:32:32.830 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:32:32.831 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:32:32.831 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:32:32.831 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:32:32.831 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:32:32.831 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:32:32.832 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:32:33.272 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:32:33.272 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:32:33.273 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:32:33.273 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:32:33.843 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:32:33.843 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:32:33.843 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:32:33.843 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:32:33.843 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:32:33.843 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:32:33.843 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:32:33.844 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:32:33.844 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:32:34.421 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:32:34.421 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:32:34.421 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-15T20:32:34.422 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:32:34.422 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-15T20:32:36.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:32:36.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:32:36.031 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:32:36.031 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:32:36.031 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:32:36.031 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:32:36.031 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:32:36.031 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:32:36.452 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:32:36.452 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:32:36.453 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:32:36.453 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:32:37.005 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:32:37.005 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:32:37.005 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:32:37.005 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:32:37.006 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:32:37.006 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:32:37.006 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:32:37.006 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:32:37.006 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:32:37.573 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:32:37.573 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:32:37.573 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-15T20:32:37.573 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:32:37.573 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-15T20:32:40.774 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:32:40.774 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:32:40.775 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:32:40.775 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:32:40.776 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:32:40.776 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:32:40.776 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:32:40.776 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:32:41.217 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:32:41.217 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:32:41.217 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:32:41.218 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:32:41.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:32:41.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:32:41.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:32:41.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:32:41.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:32:41.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:32:41.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:32:41.776 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:32:41.776 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:32:42.332 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:32:42.332 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:32:42.332 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-15T20:32:42.332 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:32:42.333 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-15T20:32:48.733 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:32:48.734 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:32:48.735 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:32:48.735 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:32:48.735 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:32:48.735 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:32:48.735 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:32:48.735 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:32:49.181 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:32:49.181 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:32:49.181 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:32:49.181 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:32:49.766 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:32:49.767 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:32:49.767 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:32:49.767 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:32:49.767 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:32:49.767 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:32:49.767 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:32:49.767 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:32:49.768 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:32:50.332 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:32:50.332 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:32:50.332 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-15T20:32:50.332 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:32:50.332 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-15T20:32:57.633 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:32:57.633 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:32:57.633 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:32:57.634 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:32:57.634 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:32:57.634 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:32:57.634 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:32:57.635 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:32:58.074 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:32:58.075 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:32:58.075 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:32:58.075 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:32:58.634 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:32:58.634 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:32:58.634 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:32:58.634 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:32:58.635 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:32:58.635 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:32:58.635 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:32:58.635 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:32:58.635 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:32:59.219 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:32:59.219 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:32:59.219 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-15T20:32:59.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-15T20:32:59.759 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:32:59.759 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_fingerprint": "75d6c3f0-55f0-4f53-878f-24c19012eb4d", 2024-09-15T20:32:59.759 INFO:tasks.workunit.client.0.smithi139.stdout: "version": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:32:59.759 INFO:tasks.workunit.client.0.smithi139.stdout: "commit": "fc3e0bff7468fd55478adbae7f75508b749124b6", 2024-09-15T20:32:59.759 INFO:tasks.workunit.client.0.smithi139.stdout: "timestamp": "2024-09-15T20:32:59.758805+0000", 2024-09-15T20:32:59.759 INFO:tasks.workunit.client.0.smithi139.stdout: "tag": "", 2024-09-15T20:32:59.759 INFO:tasks.workunit.client.0.smithi139.stdout: "health": { 2024-09-15T20:32:59.760 INFO:tasks.workunit.client.0.smithi139.stdout: "status": "HEALTH_WARN", 2024-09-15T20:32:59.760 INFO:tasks.workunit.client.0.smithi139.stdout: "checks": { 2024-09-15T20:32:59.760 INFO:tasks.workunit.client.0.smithi139.stdout: "OSDMAP_FLAGS": { 2024-09-15T20:32:59.760 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:32:59.760 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:32:59.760 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-15T20:32:59.760 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 17 2024-09-15T20:32:59.760 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.760 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [], 2024-09-15T20:32:59.760 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:32:59.760 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.760 INFO:tasks.workunit.client.0.smithi139.stdout: "OSD_DOWN": { 2024-09-15T20:32:59.760 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:32:59.761 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:32:59.761 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "1 osds down", 2024-09-15T20:32:59.761 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 1 2024-09-15T20:32:59.761 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.761 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:32:59.761 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.761 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "osd.1 (root=default,host=smithi139) is down" 2024-09-15T20:32:59.761 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.761 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:32:59.761 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:32:59.761 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.761 INFO:tasks.workunit.client.0.smithi139.stdout: "PG_DEGRADED": { 2024-09-15T20:32:59.762 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:32:59.762 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:32:59.762 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-09-15T20:32:59.762 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 1 2024-09-15T20:32:59.762 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.762 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:32:59.762 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.762 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-15T20:32:59.762 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.762 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:32:59.762 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:32:59.762 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.762 INFO:tasks.workunit.client.0.smithi139.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-15T20:32:59.763 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:32:59.763 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:32:59.763 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-15T20:32:59.763 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 1 2024-09-15T20:32:59.763 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.763 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:32:59.763 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.763 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "application not enabled on pool 'test'" 2024-09-15T20:32:59.763 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.763 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.763 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-15T20:32:59.763 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.763 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:32:59.764 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:32:59.764 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.764 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.764 INFO:tasks.workunit.client.0.smithi139.stdout: "mutes": [] 2024-09-15T20:32:59.764 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.764 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap_first_committed": 1, 2024-09-15T20:32:59.764 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap_last_committed": 1, 2024-09-15T20:32:59.764 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap": { 2024-09-15T20:32:59.764 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 1, 2024-09-15T20:32:59.764 INFO:tasks.workunit.client.0.smithi139.stdout: "fsid": "0e220df1-ac9c-433e-8885-cf9f7aab293e", 2024-09-15T20:32:59.764 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:29:28.018156Z", 2024-09-15T20:32:59.764 INFO:tasks.workunit.client.0.smithi139.stdout: "created": "2024-09-15T20:29:28.018156Z", 2024-09-15T20:32:59.764 INFO:tasks.workunit.client.0.smithi139.stdout: "min_mon_release": 19, 2024-09-15T20:32:59.765 INFO:tasks.workunit.client.0.smithi139.stdout: "min_mon_release_name": "squid", 2024-09-15T20:32:59.765 INFO:tasks.workunit.client.0.smithi139.stdout: "election_strategy": 1, 2024-09-15T20:32:59.765 INFO:tasks.workunit.client.0.smithi139.stdout: "disallowed_leaders: ": "", 2024-09-15T20:32:59.765 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode": false, 2024-09-15T20:32:59.765 INFO:tasks.workunit.client.0.smithi139.stdout: "tiebreaker_mon": "", 2024-09-15T20:32:59.765 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_ranks: ": "", 2024-09-15T20:32:59.765 INFO:tasks.workunit.client.0.smithi139.stdout: "features": { 2024-09-15T20:32:59.765 INFO:tasks.workunit.client.0.smithi139.stdout: "persistent": [ 2024-09-15T20:32:59.765 INFO:tasks.workunit.client.0.smithi139.stdout: "kraken", 2024-09-15T20:32:59.765 INFO:tasks.workunit.client.0.smithi139.stdout: "luminous", 2024-09-15T20:32:59.765 INFO:tasks.workunit.client.0.smithi139.stdout: "mimic", 2024-09-15T20:32:59.765 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap-prune", 2024-09-15T20:32:59.765 INFO:tasks.workunit.client.0.smithi139.stdout: "nautilus", 2024-09-15T20:32:59.765 INFO:tasks.workunit.client.0.smithi139.stdout: "octopus", 2024-09-15T20:32:59.766 INFO:tasks.workunit.client.0.smithi139.stdout: "pacific", 2024-09-15T20:32:59.766 INFO:tasks.workunit.client.0.smithi139.stdout: "elector-pinging", 2024-09-15T20:32:59.766 INFO:tasks.workunit.client.0.smithi139.stdout: "quincy", 2024-09-15T20:32:59.766 INFO:tasks.workunit.client.0.smithi139.stdout: "reef", 2024-09-15T20:32:59.766 INFO:tasks.workunit.client.0.smithi139.stdout: "squid" 2024-09-15T20:32:59.766 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:32:59.766 INFO:tasks.workunit.client.0.smithi139.stdout: "optional": [] 2024-09-15T20:32:59.766 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.766 INFO:tasks.workunit.client.0.smithi139.stdout: "mons": [ 2024-09-15T20:32:59.766 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.766 INFO:tasks.workunit.client.0.smithi139.stdout: "rank": 0, 2024-09-15T20:32:59.766 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "a", 2024-09-15T20:32:59.766 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:32:59.767 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:32:59.767 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.767 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:32:59.767 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:7115", 2024-09-15T20:32:59.767 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 0 2024-09-15T20:32:59.767 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.767 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.767 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.767 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:7115/0", 2024-09-15T20:32:59.767 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-15T20:32:59.767 INFO:tasks.workunit.client.0.smithi139.stdout: "priority": 0, 2024-09-15T20:32:59.767 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 0, 2024-09-15T20:32:59.767 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_location": "{}" 2024-09-15T20:32:59.767 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.768 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.768 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.768 INFO:tasks.workunit.client.0.smithi139.stdout: "quorum": [ 2024-09-15T20:32:59.768 INFO:tasks.workunit.client.0.smithi139.stdout: 0 2024-09-15T20:32:59.768 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:32:59.768 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap": { 2024-09-15T20:32:59.768 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 42, 2024-09-15T20:32:59.768 INFO:tasks.workunit.client.0.smithi139.stdout: "fsid": "0e220df1-ac9c-433e-8885-cf9f7aab293e", 2024-09-15T20:32:59.768 INFO:tasks.workunit.client.0.smithi139.stdout: "created": "2024-09-15T20:29:28.963686+0000", 2024-09-15T20:32:59.768 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:32:21.403941+0000", 2024-09-15T20:32:59.768 INFO:tasks.workunit.client.0.smithi139.stdout: "last_up_change": "2024-09-15T20:32:19.646667+0000", 2024-09-15T20:32:59.768 INFO:tasks.workunit.client.0.smithi139.stdout: "last_in_change": "2024-09-15T20:32:21.135530+0000", 2024-09-15T20:32:59.768 INFO:tasks.workunit.client.0.smithi139.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T20:32:59.768 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_num": 5800384, 2024-09-15T20:32:59.769 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_set": [ 2024-09-15T20:32:59.769 INFO:tasks.workunit.client.0.smithi139.stdout: "nodown", 2024-09-15T20:32:59.769 INFO:tasks.workunit.client.0.smithi139.stdout: "noin", 2024-09-15T20:32:59.769 INFO:tasks.workunit.client.0.smithi139.stdout: "noout", 2024-09-15T20:32:59.769 INFO:tasks.workunit.client.0.smithi139.stdout: "pglog_hardlimit", 2024-09-15T20:32:59.769 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snapdirs", 2024-09-15T20:32:59.769 INFO:tasks.workunit.client.0.smithi139.stdout: "recovery_deletes", 2024-09-15T20:32:59.769 INFO:tasks.workunit.client.0.smithi139.stdout: "sortbitwise" 2024-09-15T20:32:59.769 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:32:59.769 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_version": 9, 2024-09-15T20:32:59.769 INFO:tasks.workunit.client.0.smithi139.stdout: "full_ratio": 0.99000000953674316, 2024-09-15T20:32:59.769 INFO:tasks.workunit.client.0.smithi139.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-15T20:32:59.769 INFO:tasks.workunit.client.0.smithi139.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T20:32:59.770 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_snapshot": "", 2024-09-15T20:32:59.770 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_max": 1, 2024-09-15T20:32:59.770 INFO:tasks.workunit.client.0.smithi139.stdout: "max_osd": 3, 2024-09-15T20:32:59.770 INFO:tasks.workunit.client.0.smithi139.stdout: "require_min_compat_client": "luminous", 2024-09-15T20:32:59.770 INFO:tasks.workunit.client.0.smithi139.stdout: "min_compat_client": "jewel", 2024-09-15T20:32:59.770 INFO:tasks.workunit.client.0.smithi139.stdout: "require_osd_release": "squid", 2024-09-15T20:32:59.770 INFO:tasks.workunit.client.0.smithi139.stdout: "allow_crimson": false, 2024-09-15T20:32:59.770 INFO:tasks.workunit.client.0.smithi139.stdout: "pools": [ 2024-09-15T20:32:59.770 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.770 INFO:tasks.workunit.client.0.smithi139.stdout: "pool": 1, 2024-09-15T20:32:59.770 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_name": "test", 2024-09-15T20:32:59.770 INFO:tasks.workunit.client.0.smithi139.stdout: "create_time": "2024-09-15T20:31:19.939932+0000", 2024-09-15T20:32:59.770 INFO:tasks.workunit.client.0.smithi139.stdout: "flags": 1, 2024-09-15T20:32:59.771 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_names": "hashpspool", 2024-09-15T20:32:59.771 INFO:tasks.workunit.client.0.smithi139.stdout: "type": 1, 2024-09-15T20:32:59.771 INFO:tasks.workunit.client.0.smithi139.stdout: "size": 3, 2024-09-15T20:32:59.771 INFO:tasks.workunit.client.0.smithi139.stdout: "min_size": 2, 2024-09-15T20:32:59.771 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_rule": 0, 2024-09-15T20:32:59.771 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_count": 0, 2024-09-15T20:32:59.771 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_target": 0, 2024-09-15T20:32:59.771 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T20:32:59.771 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T20:32:59.771 INFO:tasks.workunit.client.0.smithi139.stdout: "is_stretch_pool": false, 2024-09-15T20:32:59.771 INFO:tasks.workunit.client.0.smithi139.stdout: "object_hash": 2, 2024-09-15T20:32:59.771 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_autoscale_mode": "off", 2024-09-15T20:32:59.771 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num": 1, 2024-09-15T20:32:59.772 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_placement_num": 1, 2024-09-15T20:32:59.772 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_placement_num_target": 1, 2024-09-15T20:32:59.772 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num_target": 1, 2024-09-15T20:32:59.772 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num_pending": 1, 2024-09-15T20:32:59.772 INFO:tasks.workunit.client.0.smithi139.stdout: "last_pg_merge_meta": { 2024-09-15T20:32:59.772 INFO:tasks.workunit.client.0.smithi139.stdout: "source_pgid": "0.0", 2024-09-15T20:32:59.772 INFO:tasks.workunit.client.0.smithi139.stdout: "ready_epoch": 0, 2024-09-15T20:32:59.772 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_started": 0, 2024-09-15T20:32:59.772 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_clean": 0, 2024-09-15T20:32:59.772 INFO:tasks.workunit.client.0.smithi139.stdout: "source_version": "0'0", 2024-09-15T20:32:59.772 INFO:tasks.workunit.client.0.smithi139.stdout: "target_version": "0'0" 2024-09-15T20:32:59.772 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.773 INFO:tasks.workunit.client.0.smithi139.stdout: "last_change": "29", 2024-09-15T20:32:59.773 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend": "0", 2024-09-15T20:32:59.773 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T20:32:59.773 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T20:32:59.773 INFO:tasks.workunit.client.0.smithi139.stdout: "auid": 0, 2024-09-15T20:32:59.773 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_mode": "selfmanaged", 2024-09-15T20:32:59.773 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_seq": 0, 2024-09-15T20:32:59.773 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_epoch": 0, 2024-09-15T20:32:59.773 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_snaps": [], 2024-09-15T20:32:59.773 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_snaps": "[]", 2024-09-15T20:32:59.773 INFO:tasks.workunit.client.0.smithi139.stdout: "quota_max_bytes": 0, 2024-09-15T20:32:59.773 INFO:tasks.workunit.client.0.smithi139.stdout: "quota_max_objects": 0, 2024-09-15T20:32:59.773 INFO:tasks.workunit.client.0.smithi139.stdout: "tiers": [], 2024-09-15T20:32:59.773 INFO:tasks.workunit.client.0.smithi139.stdout: "tier_of": -1, 2024-09-15T20:32:59.774 INFO:tasks.workunit.client.0.smithi139.stdout: "read_tier": -1, 2024-09-15T20:32:59.774 INFO:tasks.workunit.client.0.smithi139.stdout: "write_tier": -1, 2024-09-15T20:32:59.774 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_mode": "none", 2024-09-15T20:32:59.774 INFO:tasks.workunit.client.0.smithi139.stdout: "target_max_bytes": 0, 2024-09-15T20:32:59.774 INFO:tasks.workunit.client.0.smithi139.stdout: "target_max_objects": 0, 2024-09-15T20:32:59.774 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T20:32:59.774 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T20:32:59.774 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T20:32:59.774 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_min_flush_age": 0, 2024-09-15T20:32:59.774 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_min_evict_age": 0, 2024-09-15T20:32:59.774 INFO:tasks.workunit.client.0.smithi139.stdout: "erasure_code_profile": "", 2024-09-15T20:32:59.774 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_params": { 2024-09-15T20:32:59.774 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "none" 2024-09-15T20:32:59.774 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.775 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_period": 0, 2024-09-15T20:32:59.775 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_count": 0, 2024-09-15T20:32:59.775 INFO:tasks.workunit.client.0.smithi139.stdout: "use_gmt_hitset": true, 2024-09-15T20:32:59.775 INFO:tasks.workunit.client.0.smithi139.stdout: "min_read_recency_for_promote": 0, 2024-09-15T20:32:59.775 INFO:tasks.workunit.client.0.smithi139.stdout: "min_write_recency_for_promote": 0, 2024-09-15T20:32:59.775 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T20:32:59.775 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_search_last_n": 0, 2024-09-15T20:32:59.775 INFO:tasks.workunit.client.0.smithi139.stdout: "grade_table": [], 2024-09-15T20:32:59.775 INFO:tasks.workunit.client.0.smithi139.stdout: "stripe_width": 0, 2024-09-15T20:32:59.775 INFO:tasks.workunit.client.0.smithi139.stdout: "expected_num_objects": 0, 2024-09-15T20:32:59.775 INFO:tasks.workunit.client.0.smithi139.stdout: "fast_read": false, 2024-09-15T20:32:59.775 INFO:tasks.workunit.client.0.smithi139.stdout: "options": {}, 2024-09-15T20:32:59.775 INFO:tasks.workunit.client.0.smithi139.stdout: "application_metadata": {}, 2024-09-15T20:32:59.776 INFO:tasks.workunit.client.0.smithi139.stdout: "read_balance": { 2024-09-15T20:32:59.776 INFO:tasks.workunit.client.0.smithi139.stdout: "score_type": "Fair distribution", 2024-09-15T20:32:59.776 INFO:tasks.workunit.client.0.smithi139.stdout: "score_acting": 2.9900000095367432, 2024-09-15T20:32:59.776 INFO:tasks.workunit.client.0.smithi139.stdout: "score_stable": 2.9900000095367432, 2024-09-15T20:32:59.776 INFO:tasks.workunit.client.0.smithi139.stdout: "optimal_score": 0.67000001668930054, 2024-09-15T20:32:59.776 INFO:tasks.workunit.client.0.smithi139.stdout: "raw_score_acting": 2, 2024-09-15T20:32:59.776 INFO:tasks.workunit.client.0.smithi139.stdout: "raw_score_stable": 2, 2024-09-15T20:32:59.776 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity_weighted": 1, 2024-09-15T20:32:59.776 INFO:tasks.workunit.client.0.smithi139.stdout: "average_primary_affinity": 1, 2024-09-15T20:32:59.776 INFO:tasks.workunit.client.0.smithi139.stdout: "average_primary_affinity_weighted": 1 2024-09-15T20:32:59.776 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.776 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.776 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:32:59.776 INFO:tasks.workunit.client.0.smithi139.stdout: "osds": [ 2024-09-15T20:32:59.777 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.777 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:32:59.777 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "457b5e59-ab0b-4ff7-8ba7-80b5ae6b10f2", 2024-09-15T20:32:59.777 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 1, 2024-09-15T20:32:59.777 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:32:59.777 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:32:59.777 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:32:59.777 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 5, 2024-09-15T20:32:59.777 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 29, 2024-09-15T20:32:59.777 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 36, 2024-09-15T20:32:59.777 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 41, 2024-09-15T20:32:59.777 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 30, 2024-09-15T20:32:59.777 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:32:59.777 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:32:59.778 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:32:59.778 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.778 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:32:59.778 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6802", 2024-09-15T20:32:59.778 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1803718560 2024-09-15T20:32:59.778 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.778 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.778 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:32:59.778 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6803", 2024-09-15T20:32:59.778 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1803718560 2024-09-15T20:32:59.778 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.778 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.778 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.779 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:32:59.779 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:32:59.779 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.779 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:32:59.779 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6804", 2024-09-15T20:32:59.779 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1803718560 2024-09-15T20:32:59.779 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.779 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.779 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:32:59.779 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6805", 2024-09-15T20:32:59.779 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1803718560 2024-09-15T20:32:59.779 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.779 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.779 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.780 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:32:59.780 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:32:59.780 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.780 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:32:59.780 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6808", 2024-09-15T20:32:59.780 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1803718560 2024-09-15T20:32:59.780 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.780 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.780 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:32:59.780 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6809", 2024-09-15T20:32:59.780 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1803718560 2024-09-15T20:32:59.780 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.780 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.781 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.781 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:32:59.781 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:32:59.781 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.781 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:32:59.781 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6806", 2024-09-15T20:32:59.781 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1803718560 2024-09-15T20:32:59.781 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.781 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.781 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:32:59.781 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6807", 2024-09-15T20:32:59.781 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1803718560 2024-09-15T20:32:59.781 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.781 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.782 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.782 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6803/1803718560", 2024-09-15T20:32:59.782 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6805/1803718560", 2024-09-15T20:32:59.782 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1803718560", 2024-09-15T20:32:59.782 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1803718560", 2024-09-15T20:32:59.782 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:32:59.782 INFO:tasks.workunit.client.0.smithi139.stdout: "exists", 2024-09-15T20:32:59.782 INFO:tasks.workunit.client.0.smithi139.stdout: "up" 2024-09-15T20:32:59.782 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.782 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.782 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.782 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:32:59.782 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "037d24c1-96ec-4453-b5af-a66b6de4e720", 2024-09-15T20:32:59.783 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 0, 2024-09-15T20:32:59.783 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:32:59.783 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:32:59.783 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:32:59.783 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 0, 2024-09-15T20:32:59.783 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 0, 2024-09-15T20:32:59.783 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 10, 2024-09-15T20:32:59.783 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 23, 2024-09-15T20:32:59.783 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 31, 2024-09-15T20:32:59.783 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:32:59.783 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:32:59.783 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:32:59.783 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.783 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:32:59.784 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6810", 2024-09-15T20:32:59.784 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 4110363107 2024-09-15T20:32:59.784 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.784 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.784 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:32:59.784 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6811", 2024-09-15T20:32:59.784 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 4110363107 2024-09-15T20:32:59.784 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.784 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.784 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.784 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:32:59.784 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:32:59.784 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.784 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:32:59.785 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6812", 2024-09-15T20:32:59.785 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 4110363107 2024-09-15T20:32:59.785 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.785 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.785 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:32:59.785 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6813", 2024-09-15T20:32:59.785 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 4110363107 2024-09-15T20:32:59.785 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.785 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.785 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.785 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:32:59.785 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:32:59.785 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.785 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:32:59.786 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6816", 2024-09-15T20:32:59.786 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 4110363107 2024-09-15T20:32:59.786 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.786 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.786 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:32:59.786 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6817", 2024-09-15T20:32:59.786 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 4110363107 2024-09-15T20:32:59.786 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.786 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.786 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.786 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:32:59.786 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:32:59.786 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.786 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:32:59.787 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6814", 2024-09-15T20:32:59.787 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 4110363107 2024-09-15T20:32:59.787 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.787 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.787 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:32:59.787 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6815", 2024-09-15T20:32:59.787 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 4110363107 2024-09-15T20:32:59.787 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.787 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.787 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.787 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6811/4110363107", 2024-09-15T20:32:59.787 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6813/4110363107", 2024-09-15T20:32:59.787 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6817/4110363107", 2024-09-15T20:32:59.788 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6815/4110363107", 2024-09-15T20:32:59.788 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:32:59.788 INFO:tasks.workunit.client.0.smithi139.stdout: "exists" 2024-09-15T20:32:59.788 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.788 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.788 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.788 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:32:59.788 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "24e3e077-bb9a-4ae3-9be6-20ce6b775ae4", 2024-09-15T20:32:59.788 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 1, 2024-09-15T20:32:59.788 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:32:59.788 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:32:59.788 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:32:59.788 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 15, 2024-09-15T20:32:59.788 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 29, 2024-09-15T20:32:59.789 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 38, 2024-09-15T20:32:59.789 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 0, 2024-09-15T20:32:59.789 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 32, 2024-09-15T20:32:59.789 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:32:59.789 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:32:59.789 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:32:59.789 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.789 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:32:59.789 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6810", 2024-09-15T20:32:59.789 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3470466229 2024-09-15T20:32:59.789 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.789 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.789 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:32:59.790 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6811", 2024-09-15T20:32:59.790 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3470466229 2024-09-15T20:32:59.790 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.790 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.790 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.790 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:32:59.790 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:32:59.790 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.790 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:32:59.790 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6812", 2024-09-15T20:32:59.790 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3470466229 2024-09-15T20:32:59.790 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.790 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.790 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:32:59.791 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6813", 2024-09-15T20:32:59.791 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3470466229 2024-09-15T20:32:59.791 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.791 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.791 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.791 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:32:59.791 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:32:59.791 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.791 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:32:59.791 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6816", 2024-09-15T20:32:59.791 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3470466229 2024-09-15T20:32:59.791 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.791 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.791 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:32:59.792 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6817", 2024-09-15T20:32:59.792 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3470466229 2024-09-15T20:32:59.792 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.792 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.792 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.792 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:32:59.792 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:32:59.792 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.792 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:32:59.792 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6814", 2024-09-15T20:32:59.792 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3470466229 2024-09-15T20:32:59.792 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.792 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.793 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:32:59.793 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6815", 2024-09-15T20:32:59.793 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3470466229 2024-09-15T20:32:59.793 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.793 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.793 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.793 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6811/3470466229", 2024-09-15T20:32:59.793 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6813/3470466229", 2024-09-15T20:32:59.793 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3470466229", 2024-09-15T20:32:59.793 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3470466229", 2024-09-15T20:32:59.793 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:32:59.793 INFO:tasks.workunit.client.0.smithi139.stdout: "exists", 2024-09-15T20:32:59.793 INFO:tasks.workunit.client.0.smithi139.stdout: "up" 2024-09-15T20:32:59.794 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.794 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.794 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:32:59.794 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_xinfo": [ 2024-09-15T20:32:59.794 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.794 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:32:59.794 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:32:06.380843+0000", 2024-09-15T20:32:59.794 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:32:59.794 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:32:59.794 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:32:59.794 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:32:59.794 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:30:04.676305+0000", 2024-09-15T20:32:59.795 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:32:59.795 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.795 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.795 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:32:59.795 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:32:06.958649+0000", 2024-09-15T20:32:59.795 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:32:59.795 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:32:59.795 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:32:59.795 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:32:59.795 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:30:40.276160+0000", 2024-09-15T20:32:59.795 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:32:59.795 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.795 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.795 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:32:59.796 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:32:07.595692+0000", 2024-09-15T20:32:59.796 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:32:59.796 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:32:59.796 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:32:59.796 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:32:59.796 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:31:15.824964+0000", 2024-09-15T20:32:59.796 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:32:59.796 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.796 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:32:59.796 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap": [], 2024-09-15T20:32:59.796 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap_items": [], 2024-09-15T20:32:59.796 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap_primaries": [], 2024-09-15T20:32:59.796 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_temp": [], 2024-09-15T20:32:59.796 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_temp": [], 2024-09-15T20:32:59.797 INFO:tasks.workunit.client.0.smithi139.stdout: "blocklist": {}, 2024-09-15T20:32:59.797 INFO:tasks.workunit.client.0.smithi139.stdout: "range_blocklist": {}, 2024-09-15T20:32:59.797 INFO:tasks.workunit.client.0.smithi139.stdout: "erasure_code_profiles": { 2024-09-15T20:32:59.797 INFO:tasks.workunit.client.0.smithi139.stdout: "default": { 2024-09-15T20:32:59.797 INFO:tasks.workunit.client.0.smithi139.stdout: "k": "2", 2024-09-15T20:32:59.797 INFO:tasks.workunit.client.0.smithi139.stdout: "m": "2", 2024-09-15T20:32:59.797 INFO:tasks.workunit.client.0.smithi139.stdout: "plugin": "jerasure", 2024-09-15T20:32:59.797 INFO:tasks.workunit.client.0.smithi139.stdout: "technique": "reed_sol_van" 2024-09-15T20:32:59.797 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.797 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.797 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_snaps_queue": [], 2024-09-15T20:32:59.797 INFO:tasks.workunit.client.0.smithi139.stdout: "new_removed_snaps": [], 2024-09-15T20:32:59.797 INFO:tasks.workunit.client.0.smithi139.stdout: "new_purged_snaps": [], 2024-09-15T20:32:59.797 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_node_flags": {}, 2024-09-15T20:32:59.798 INFO:tasks.workunit.client.0.smithi139.stdout: "device_class_flags": {}, 2024-09-15T20:32:59.798 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode": { 2024-09-15T20:32:59.798 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode_enabled": false, 2024-09-15T20:32:59.798 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_bucket_count": 0, 2024-09-15T20:32:59.798 INFO:tasks.workunit.client.0.smithi139.stdout: "degraded_stretch_mode": 0, 2024-09-15T20:32:59.798 INFO:tasks.workunit.client.0.smithi139.stdout: "recovering_stretch_mode": 0, 2024-09-15T20:32:59.798 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode_bucket": 0 2024-09-15T20:32:59.798 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.798 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.798 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_metadata": [ 2024-09-15T20:32:59.798 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.798 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:32:59.798 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:32:59.798 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6804/1803718560,v1:127.0.0.1:6805/1803718560]", 2024-09-15T20:32:59.799 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:32:59.799 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:32:59.799 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:32:59.799 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:32:59.799 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:32:59.799 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:32:59.799 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:32:59.799 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:32:59.799 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:32:59.799 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:32:59.799 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:32:59.799 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-15T20:32:59.799 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:32:59.799 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:32:59.800 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:32:59.800 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:32:59.800 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:32:59.800 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:32:59.800 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:32:59.800 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:32:59.800 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:32:59.800 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:32:59.800 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:29:35.554331Z", 2024-09-15T20:32:59.800 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:32:59.800 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:32:59.800 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:32:59.800 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:32:59.800 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:32:59.801 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:32:59.801 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:32:59.801 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6802/1803718560,v1:127.0.0.1:6803/1803718560]", 2024-09-15T20:32:59.801 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:32:59.801 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1803718560,v1:127.0.0.1:6809/1803718560]", 2024-09-15T20:32:59.801 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1803718560,v1:127.0.0.1:6807/1803718560]", 2024-09-15T20:32:59.801 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:32:59.801 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:32:59.801 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:32:59.801 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:32:59.801 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:32:59.801 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:32:59.801 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:32:59.802 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:32:59.802 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:32:59.802 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/0", 2024-09-15T20:32:59.802 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:32:59.802 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:32:59.802 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:32:59.802 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.802 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.802 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:32:59.802 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:32:59.802 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6812/4110363107,v1:127.0.0.1:6813/4110363107]", 2024-09-15T20:32:59.802 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:32:59.802 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:32:59.802 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:32:59.803 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:32:59.803 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:32:59.803 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:32:59.803 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:32:59.803 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:32:59.803 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:32:59.803 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:32:59.803 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:32:59.803 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-15T20:32:59.803 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:32:59.803 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:32:59.803 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:32:59.803 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:32:59.803 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:32:59.804 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:32:59.804 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:32:59.804 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:32:59.804 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:32:59.804 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:32:59.804 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:30:12.116512Z", 2024-09-15T20:32:59.804 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:32:59.804 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:32:59.804 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:32:59.804 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:32:59.805 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:32:59.805 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:32:59.805 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:32:59.805 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6810/4110363107,v1:127.0.0.1:6811/4110363107]", 2024-09-15T20:32:59.805 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:32:59.805 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/4110363107,v1:127.0.0.1:6817/4110363107]", 2024-09-15T20:32:59.805 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/4110363107,v1:127.0.0.1:6815/4110363107]", 2024-09-15T20:32:59.805 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:32:59.805 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:32:59.805 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:32:59.805 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:32:59.805 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:32:59.805 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:32:59.806 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:32:59.806 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:32:59.806 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:32:59.806 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/1", 2024-09-15T20:32:59.806 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:32:59.806 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:32:59.806 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:32:59.806 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.806 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.806 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:32:59.806 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:32:59.806 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6812/3470466229,v1:127.0.0.1:6813/3470466229]", 2024-09-15T20:32:59.806 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:32:59.806 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:32:59.807 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:32:59.807 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:32:59.807 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:32:59.807 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:32:59.807 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:32:59.807 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:32:59.807 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:32:59.807 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:32:59.807 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:32:59.807 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-15T20:32:59.807 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:32:59.807 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:32:59.807 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:32:59.807 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:32:59.808 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:32:59.808 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:32:59.808 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:32:59.808 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:32:59.808 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:32:59.808 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:32:59.808 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:30:47.443148Z", 2024-09-15T20:32:59.808 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:32:59.808 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:32:59.808 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:32:59.808 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:32:59.808 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:32:59.808 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:32:59.808 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:32:59.809 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6810/3470466229,v1:127.0.0.1:6811/3470466229]", 2024-09-15T20:32:59.809 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:32:59.809 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3470466229,v1:127.0.0.1:6817/3470466229]", 2024-09-15T20:32:59.809 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3470466229,v1:127.0.0.1:6815/3470466229]", 2024-09-15T20:32:59.809 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:32:59.809 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:32:59.809 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:32:59.809 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:32:59.809 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:32:59.809 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:32:59.809 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:32:59.809 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:32:59.810 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:32:59.810 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/2", 2024-09-15T20:32:59.810 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:32:59.810 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:32:59.810 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:32:59.810 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.810 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:32:59.810 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_clean_epochs": { 2024-09-15T20:32:59.810 INFO:tasks.workunit.client.0.smithi139.stdout: "min_last_epoch_clean": 0, 2024-09-15T20:32:59.810 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_clean": { 2024-09-15T20:32:59.810 INFO:tasks.workunit.client.0.smithi139.stdout: "per_pool": [] 2024-09-15T20:32:59.811 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.811 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_epochs": [ 2024-09-15T20:32:59.811 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.811 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:32:59.811 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 36 2024-09-15T20:32:59.811 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.811 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.811 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:32:59.811 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 38 2024-09-15T20:32:59.811 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.811 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.811 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.811 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_first_committed": 1, 2024-09-15T20:32:59.812 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_last_committed": 42, 2024-09-15T20:32:59.812 INFO:tasks.workunit.client.0.smithi139.stdout: "crushmap": { 2024-09-15T20:32:59.812 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": [ 2024-09-15T20:32:59.812 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.812 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:32:59.812 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.0", 2024-09-15T20:32:59.812 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:32:59.812 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.812 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.812 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:32:59.812 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.1", 2024-09-15T20:32:59.812 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:32:59.812 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.813 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.813 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:32:59.813 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.2", 2024-09-15T20:32:59.813 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:32:59.813 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.813 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:32:59.813 INFO:tasks.workunit.client.0.smithi139.stdout: "types": [ 2024-09-15T20:32:59.813 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.813 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 0, 2024-09-15T20:32:59.813 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd" 2024-09-15T20:32:59.813 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.813 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.813 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:32:59.814 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "host" 2024-09-15T20:32:59.814 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.814 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.814 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 2, 2024-09-15T20:32:59.814 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "chassis" 2024-09-15T20:32:59.814 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.814 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.814 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 3, 2024-09-15T20:32:59.814 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "rack" 2024-09-15T20:32:59.814 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.814 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.814 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 4, 2024-09-15T20:32:59.814 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "row" 2024-09-15T20:32:59.814 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.815 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.815 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 5, 2024-09-15T20:32:59.815 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "pdu" 2024-09-15T20:32:59.815 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.815 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.815 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 6, 2024-09-15T20:32:59.815 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "pod" 2024-09-15T20:32:59.815 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.815 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.815 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 7, 2024-09-15T20:32:59.815 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "room" 2024-09-15T20:32:59.815 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.815 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.815 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 8, 2024-09-15T20:32:59.816 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "datacenter" 2024-09-15T20:32:59.816 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.816 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.816 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 9, 2024-09-15T20:32:59.816 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "zone" 2024-09-15T20:32:59.816 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.816 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.816 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 10, 2024-09-15T20:32:59.816 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "region" 2024-09-15T20:32:59.816 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.816 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.816 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:32:59.816 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "root" 2024-09-15T20:32:59.817 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.817 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:32:59.817 INFO:tasks.workunit.client.0.smithi139.stdout: "buckets": [ 2024-09-15T20:32:59.817 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.817 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -1, 2024-09-15T20:32:59.817 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "default", 2024-09-15T20:32:59.817 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:32:59.817 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "root", 2024-09-15T20:32:59.817 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:32:59.817 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:32:59.817 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:32:59.817 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:32:59.817 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.817 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -3, 2024-09-15T20:32:59.818 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:32:59.818 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:32:59.818 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.818 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.818 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.818 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.818 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -2, 2024-09-15T20:32:59.818 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "default~hdd", 2024-09-15T20:32:59.818 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:32:59.818 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "root", 2024-09-15T20:32:59.818 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:32:59.818 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:32:59.818 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:32:59.819 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:32:59.819 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.819 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -4, 2024-09-15T20:32:59.819 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:32:59.819 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:32:59.819 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.819 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.819 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.819 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.819 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -3, 2024-09-15T20:32:59.819 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "smithi139", 2024-09-15T20:32:59.819 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:32:59.819 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "host", 2024-09-15T20:32:59.820 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:32:59.820 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:32:59.820 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:32:59.820 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:32:59.820 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.820 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:32:59.820 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:32:59.820 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:32:59.820 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.820 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.820 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:32:59.820 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:32:59.820 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 1 2024-09-15T20:32:59.820 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.821 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.821 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:32:59.821 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:32:59.821 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 2 2024-09-15T20:32:59.821 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.821 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.821 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.821 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.821 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -4, 2024-09-15T20:32:59.821 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "smithi139~hdd", 2024-09-15T20:32:59.821 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:32:59.821 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "host", 2024-09-15T20:32:59.821 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:32:59.821 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:32:59.822 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:32:59.822 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:32:59.822 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.822 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:32:59.822 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:32:59.822 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:32:59.822 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.822 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.822 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:32:59.822 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:32:59.822 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 1 2024-09-15T20:32:59.822 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.822 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.822 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:32:59.823 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:32:59.823 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 2 2024-09-15T20:32:59.823 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.823 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.823 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.823 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:32:59.823 INFO:tasks.workunit.client.0.smithi139.stdout: "rules": [ 2024-09-15T20:32:59.823 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.823 INFO:tasks.workunit.client.0.smithi139.stdout: "rule_id": 0, 2024-09-15T20:32:59.823 INFO:tasks.workunit.client.0.smithi139.stdout: "rule_name": "replicated_rule", 2024-09-15T20:32:59.823 INFO:tasks.workunit.client.0.smithi139.stdout: "type": 1, 2024-09-15T20:32:59.823 INFO:tasks.workunit.client.0.smithi139.stdout: "steps": [ 2024-09-15T20:32:59.823 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.823 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "take", 2024-09-15T20:32:59.824 INFO:tasks.workunit.client.0.smithi139.stdout: "item": -1, 2024-09-15T20:32:59.824 INFO:tasks.workunit.client.0.smithi139.stdout: "item_name": "default" 2024-09-15T20:32:59.824 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.824 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.824 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "choose_firstn", 2024-09-15T20:32:59.824 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 0, 2024-09-15T20:32:59.824 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "osd" 2024-09-15T20:32:59.824 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.824 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.824 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "emit" 2024-09-15T20:32:59.824 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.824 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:32:59.824 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.824 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:32:59.825 INFO:tasks.workunit.client.0.smithi139.stdout: "tunables": { 2024-09-15T20:32:59.825 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_local_tries": 0, 2024-09-15T20:32:59.825 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_local_fallback_tries": 0, 2024-09-15T20:32:59.825 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_total_tries": 50, 2024-09-15T20:32:59.825 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_descend_once": 1, 2024-09-15T20:32:59.825 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_vary_r": 1, 2024-09-15T20:32:59.825 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_stable": 1, 2024-09-15T20:32:59.825 INFO:tasks.workunit.client.0.smithi139.stdout: "msr_descents": 100, 2024-09-15T20:32:59.825 INFO:tasks.workunit.client.0.smithi139.stdout: "msr_collision_tries": 100, 2024-09-15T20:32:59.825 INFO:tasks.workunit.client.0.smithi139.stdout: "straw_calc_version": 1, 2024-09-15T20:32:59.825 INFO:tasks.workunit.client.0.smithi139.stdout: "allowed_bucket_algs": 54, 2024-09-15T20:32:59.825 INFO:tasks.workunit.client.0.smithi139.stdout: "profile": "jewel", 2024-09-15T20:32:59.825 INFO:tasks.workunit.client.0.smithi139.stdout: "optimal_tunables": 1, 2024-09-15T20:32:59.825 INFO:tasks.workunit.client.0.smithi139.stdout: "legacy_tunables": 0, 2024-09-15T20:32:59.825 INFO:tasks.workunit.client.0.smithi139.stdout: "minimum_required_version": "jewel", 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables": 1, 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables2": 1, 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v2_rules": 0, 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables3": 1, 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v3_rules": 0, 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v4_buckets": 1, 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables5": 1, 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v5_rules": 0, 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi139.stdout: "has_msr_rules": 0 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_args": {} 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi139.stdout: "fsmap": { 2024-09-15T20:32:59.826 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 1, 2024-09-15T20:32:59.827 INFO:tasks.workunit.client.0.smithi139.stdout: "btime": "2024-09-15T20:29:28:939760+0000", 2024-09-15T20:32:59.827 INFO:tasks.workunit.client.0.smithi139.stdout: "default_fscid": -1, 2024-09-15T20:32:59.827 INFO:tasks.workunit.client.0.smithi139.stdout: "compat": { 2024-09-15T20:32:59.827 INFO:tasks.workunit.client.0.smithi139.stdout: "compat": {}, 2024-09-15T20:32:59.827 INFO:tasks.workunit.client.0.smithi139.stdout: "ro_compat": {}, 2024-09-15T20:32:59.827 INFO:tasks.workunit.client.0.smithi139.stdout: "incompat": { 2024-09-15T20:32:59.827 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_1": "base v0.20", 2024-09-15T20:32:59.827 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_2": "client writeable ranges", 2024-09-15T20:32:59.827 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_3": "default file layouts on dirs", 2024-09-15T20:32:59.827 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_4": "dir inode in separate object", 2024-09-15T20:32:59.827 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_5": "mds uses versioned encoding", 2024-09-15T20:32:59.827 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-15T20:32:59.827 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_8": "no anchor table", 2024-09-15T20:32:59.827 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_9": "file layout v2", 2024-09-15T20:32:59.828 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_10": "snaprealm v2", 2024-09-15T20:32:59.828 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_11": "minor log segments", 2024-09-15T20:32:59.828 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_12": "quiesce subvolumes" 2024-09-15T20:32:59.828 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.828 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.828 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_flags": { 2024-09-15T20:32:59.828 INFO:tasks.workunit.client.0.smithi139.stdout: "enable_multiple": true, 2024-09-15T20:32:59.828 INFO:tasks.workunit.client.0.smithi139.stdout: "ever_enabled_multiple": true 2024-09-15T20:32:59.828 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.828 INFO:tasks.workunit.client.0.smithi139.stdout: "standbys": [], 2024-09-15T20:32:59.828 INFO:tasks.workunit.client.0.smithi139.stdout: "filesystems": [] 2024-09-15T20:32:59.828 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.828 INFO:tasks.workunit.client.0.smithi139.stdout: "mdsmap_first_committed": 1, 2024-09-15T20:32:59.828 INFO:tasks.workunit.client.0.smithi139.stdout: "mdsmap_last_committed": 1, 2024-09-15T20:32:59.829 INFO:tasks.workunit.client.0.smithi139.stdout: "auth": { 2024-09-15T20:32:59.829 INFO:tasks.workunit.client.0.smithi139.stdout: "first_committed": 1, 2024-09-15T20:32:59.829 INFO:tasks.workunit.client.0.smithi139.stdout: "last_committed": 5, 2024-09-15T20:32:59.829 INFO:tasks.workunit.client.0.smithi139.stdout: "num_secrets": 3 2024-09-15T20:32:59.829 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.829 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg": 1, 2024-09-15T20:32:59.829 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_active": 1, 2024-09-15T20:32:59.829 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_unknown": 0, 2024-09-15T20:32:59.829 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osd": 3, 2024-09-15T20:32:59.829 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_sum": { 2024-09-15T20:32:59.829 INFO:tasks.workunit.client.0.smithi139.stdout: "stat_sum": { 2024-09-15T20:32:59.829 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes": 61700, 2024-09-15T20:32:59.829 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects": 100, 2024-09-15T20:32:59.829 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_clones": 0, 2024-09-15T20:32:59.830 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_copies": 300, 2024-09-15T20:32:59.830 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T20:32:59.830 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing": 0, 2024-09-15T20:32:59.830 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_degraded": 100, 2024-09-15T20:32:59.830 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_misplaced": 0, 2024-09-15T20:32:59.830 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_unfound": 0, 2024-09-15T20:32:59.830 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_dirty": 100, 2024-09-15T20:32:59.830 INFO:tasks.workunit.client.0.smithi139.stdout: "num_whiteouts": 0, 2024-09-15T20:32:59.830 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read": 0, 2024-09-15T20:32:59.830 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read_kb": 0, 2024-09-15T20:32:59.830 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write": 100, 2024-09-15T20:32:59.830 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write_kb": 100, 2024-09-15T20:32:59.830 INFO:tasks.workunit.client.0.smithi139.stdout: "num_scrub_errors": 0, 2024-09-15T20:32:59.831 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T20:32:59.831 INFO:tasks.workunit.client.0.smithi139.stdout: "num_deep_scrub_errors": 0, 2024-09-15T20:32:59.831 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_recovered": 0, 2024-09-15T20:32:59.831 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_recovered": 0, 2024-09-15T20:32:59.831 INFO:tasks.workunit.client.0.smithi139.stdout: "num_keys_recovered": 0, 2024-09-15T20:32:59.831 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_omap": 0, 2024-09-15T20:32:59.831 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T20:32:59.831 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T20:32:59.831 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush": 0, 2024-09-15T20:32:59.831 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_kb": 0, 2024-09-15T20:32:59.831 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict": 0, 2024-09-15T20:32:59.831 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_kb": 0, 2024-09-15T20:32:59.831 INFO:tasks.workunit.client.0.smithi139.stdout: "num_promote": 0, 2024-09-15T20:32:59.831 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_high": 0, 2024-09-15T20:32:59.832 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_low": 0, 2024-09-15T20:32:59.832 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_some": 0, 2024-09-15T20:32:59.832 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_full": 0, 2024-09-15T20:32:59.832 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_pinned": 0, 2024-09-15T20:32:59.832 INFO:tasks.workunit.client.0.smithi139.stdout: "num_legacy_snapsets": 0, 2024-09-15T20:32:59.832 INFO:tasks.workunit.client.0.smithi139.stdout: "num_large_omap_objects": 0, 2024-09-15T20:32:59.832 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_manifest": 0, 2024-09-15T20:32:59.832 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_bytes": 0, 2024-09-15T20:32:59.832 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_keys": 0, 2024-09-15T20:32:59.832 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_repaired": 0 2024-09-15T20:32:59.832 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.832 INFO:tasks.workunit.client.0.smithi139.stdout: "store_stats": { 2024-09-15T20:32:59.832 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 0, 2024-09-15T20:32:59.833 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 0, 2024-09-15T20:32:59.833 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:32:59.833 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 0, 2024-09-15T20:32:59.833 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 0, 2024-09-15T20:32:59.833 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:32:59.833 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:32:59.833 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:32:59.833 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 0, 2024-09-15T20:32:59.833 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 0 2024-09-15T20:32:59.833 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.833 INFO:tasks.workunit.client.0.smithi139.stdout: "log_size": 16, 2024-09-15T20:32:59.833 INFO:tasks.workunit.client.0.smithi139.stdout: "ondisk_log_size": 16, 2024-09-15T20:32:59.833 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 2, 2024-09-15T20:32:59.834 INFO:tasks.workunit.client.0.smithi139.stdout: "acting": 2, 2024-09-15T20:32:59.834 INFO:tasks.workunit.client.0.smithi139.stdout: "num_store_stats": 0 2024-09-15T20:32:59.834 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.834 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_sum": { 2024-09-15T20:32:59.834 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 0, 2024-09-15T20:32:59.834 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 0, 2024-09-15T20:32:59.834 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pgs": 3, 2024-09-15T20:32:59.834 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osds": 3, 2024-09-15T20:32:59.834 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_osds": 3, 2024-09-15T20:32:59.834 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T20:32:59.834 INFO:tasks.workunit.client.0.smithi139.stdout: "kb": 314572800, 2024-09-15T20:32:59.834 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used": 83684, 2024-09-15T20:32:59.834 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_data": 2120, 2024-09-15T20:32:59.834 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_omap": 10, 2024-09-15T20:32:59.835 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_meta": 81461, 2024-09-15T20:32:59.835 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_avail": 314489116, 2024-09-15T20:32:59.835 INFO:tasks.workunit.client.0.smithi139.stdout: "statfs": { 2024-09-15T20:32:59.835 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 322122547200, 2024-09-15T20:32:59.835 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 322036854784, 2024-09-15T20:32:59.835 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:32:59.835 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 2170880, 2024-09-15T20:32:59.835 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 465198, 2024-09-15T20:32:59.835 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:32:59.835 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:32:59.835 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:32:59.835 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 10396, 2024-09-15T20:32:59.835 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 83416932 2024-09-15T20:32:59.835 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.836 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_peers": [], 2024-09-15T20:32:59.836 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_trim_queue_len": 0, 2024-09-15T20:32:59.836 INFO:tasks.workunit.client.0.smithi139.stdout: "num_snap_trimming": 0, 2024-09-15T20:32:59.836 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shards_repaired": 0, 2024-09-15T20:32:59.836 INFO:tasks.workunit.client.0.smithi139.stdout: "op_queue_age_hist": { 2024-09-15T20:32:59.836 INFO:tasks.workunit.client.0.smithi139.stdout: "histogram": [], 2024-09-15T20:32:59.836 INFO:tasks.workunit.client.0.smithi139.stdout: "upper_bound": 1 2024-09-15T20:32:59.836 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.836 INFO:tasks.workunit.client.0.smithi139.stdout: "perf_stat": { 2024-09-15T20:32:59.836 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ms": 0, 2024-09-15T20:32:59.836 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ms": 0, 2024-09-15T20:32:59.836 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ns": 0, 2024-09-15T20:32:59.837 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ns": 0 2024-09-15T20:32:59.837 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.837 INFO:tasks.workunit.client.0.smithi139.stdout: "alerts": [], 2024-09-15T20:32:59.837 INFO:tasks.workunit.client.0.smithi139.stdout: "network_ping_times": [] 2024-09-15T20:32:59.837 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.837 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_sum_by_class": { 2024-09-15T20:32:59.837 INFO:tasks.workunit.client.0.smithi139.stdout: "hdd": { 2024-09-15T20:32:59.837 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 0, 2024-09-15T20:32:59.837 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 0, 2024-09-15T20:32:59.837 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pgs": 3, 2024-09-15T20:32:59.837 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osds": 3, 2024-09-15T20:32:59.837 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_osds": 3, 2024-09-15T20:32:59.837 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T20:32:59.837 INFO:tasks.workunit.client.0.smithi139.stdout: "kb": 314572800, 2024-09-15T20:32:59.838 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used": 83684, 2024-09-15T20:32:59.838 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_data": 2120, 2024-09-15T20:32:59.838 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_omap": 10, 2024-09-15T20:32:59.838 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_meta": 81461, 2024-09-15T20:32:59.838 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_avail": 314489116, 2024-09-15T20:32:59.838 INFO:tasks.workunit.client.0.smithi139.stdout: "statfs": { 2024-09-15T20:32:59.838 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 322122547200, 2024-09-15T20:32:59.838 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 322036854784, 2024-09-15T20:32:59.838 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:32:59.838 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 2170880, 2024-09-15T20:32:59.838 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 465198, 2024-09-15T20:32:59.838 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:32:59.838 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:32:59.839 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:32:59.839 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 10396, 2024-09-15T20:32:59.839 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 83416932 2024-09-15T20:32:59.839 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.839 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_peers": [], 2024-09-15T20:32:59.839 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_trim_queue_len": 0, 2024-09-15T20:32:59.839 INFO:tasks.workunit.client.0.smithi139.stdout: "num_snap_trimming": 0, 2024-09-15T20:32:59.839 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shards_repaired": 0, 2024-09-15T20:32:59.839 INFO:tasks.workunit.client.0.smithi139.stdout: "op_queue_age_hist": { 2024-09-15T20:32:59.839 INFO:tasks.workunit.client.0.smithi139.stdout: "histogram": [], 2024-09-15T20:32:59.839 INFO:tasks.workunit.client.0.smithi139.stdout: "upper_bound": 1 2024-09-15T20:32:59.839 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.839 INFO:tasks.workunit.client.0.smithi139.stdout: "perf_stat": { 2024-09-15T20:32:59.839 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ms": 0, 2024-09-15T20:32:59.840 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ms": 0, 2024-09-15T20:32:59.840 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ns": 0, 2024-09-15T20:32:59.840 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ns": 0 2024-09-15T20:32:59.840 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.840 INFO:tasks.workunit.client.0.smithi139.stdout: "alerts": [], 2024-09-15T20:32:59.840 INFO:tasks.workunit.client.0.smithi139.stdout: "network_ping_times": [] 2024-09-15T20:32:59.840 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.840 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.840 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_stats": [ 2024-09-15T20:32:59.840 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.840 INFO:tasks.workunit.client.0.smithi139.stdout: "poolid": 1, 2024-09-15T20:32:59.840 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg": 1, 2024-09-15T20:32:59.840 INFO:tasks.workunit.client.0.smithi139.stdout: "stat_sum": { 2024-09-15T20:32:59.841 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes": 61700, 2024-09-15T20:32:59.841 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects": 100, 2024-09-15T20:32:59.841 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_clones": 0, 2024-09-15T20:32:59.841 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_copies": 300, 2024-09-15T20:32:59.841 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T20:32:59.841 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing": 0, 2024-09-15T20:32:59.841 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_degraded": 100, 2024-09-15T20:32:59.841 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_misplaced": 0, 2024-09-15T20:32:59.841 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_unfound": 0, 2024-09-15T20:32:59.841 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_dirty": 100, 2024-09-15T20:32:59.841 INFO:tasks.workunit.client.0.smithi139.stdout: "num_whiteouts": 0, 2024-09-15T20:32:59.841 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read": 0, 2024-09-15T20:32:59.841 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read_kb": 0, 2024-09-15T20:32:59.841 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write": 100, 2024-09-15T20:32:59.842 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write_kb": 100, 2024-09-15T20:32:59.842 INFO:tasks.workunit.client.0.smithi139.stdout: "num_scrub_errors": 0, 2024-09-15T20:32:59.842 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T20:32:59.842 INFO:tasks.workunit.client.0.smithi139.stdout: "num_deep_scrub_errors": 0, 2024-09-15T20:32:59.842 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_recovered": 0, 2024-09-15T20:32:59.842 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_recovered": 0, 2024-09-15T20:32:59.842 INFO:tasks.workunit.client.0.smithi139.stdout: "num_keys_recovered": 0, 2024-09-15T20:32:59.842 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_omap": 0, 2024-09-15T20:32:59.842 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T20:32:59.842 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T20:32:59.842 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush": 0, 2024-09-15T20:32:59.842 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_kb": 0, 2024-09-15T20:32:59.842 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict": 0, 2024-09-15T20:32:59.842 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_kb": 0, 2024-09-15T20:32:59.843 INFO:tasks.workunit.client.0.smithi139.stdout: "num_promote": 0, 2024-09-15T20:32:59.843 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_high": 0, 2024-09-15T20:32:59.843 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_low": 0, 2024-09-15T20:32:59.843 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_some": 0, 2024-09-15T20:32:59.843 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_full": 0, 2024-09-15T20:32:59.843 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_pinned": 0, 2024-09-15T20:32:59.843 INFO:tasks.workunit.client.0.smithi139.stdout: "num_legacy_snapsets": 0, 2024-09-15T20:32:59.843 INFO:tasks.workunit.client.0.smithi139.stdout: "num_large_omap_objects": 0, 2024-09-15T20:32:59.843 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_manifest": 0, 2024-09-15T20:32:59.843 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_bytes": 0, 2024-09-15T20:32:59.843 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_keys": 0, 2024-09-15T20:32:59.843 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_repaired": 0 2024-09-15T20:32:59.843 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.844 INFO:tasks.workunit.client.0.smithi139.stdout: "store_stats": { 2024-09-15T20:32:59.844 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 0, 2024-09-15T20:32:59.844 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 0, 2024-09-15T20:32:59.844 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:32:59.844 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 1220608, 2024-09-15T20:32:59.844 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 184566, 2024-09-15T20:32:59.844 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:32:59.844 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:32:59.844 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:32:59.844 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 0, 2024-09-15T20:32:59.844 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 0 2024-09-15T20:32:59.844 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.844 INFO:tasks.workunit.client.0.smithi139.stdout: "log_size": 16, 2024-09-15T20:32:59.844 INFO:tasks.workunit.client.0.smithi139.stdout: "ondisk_log_size": 16, 2024-09-15T20:32:59.845 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 2, 2024-09-15T20:32:59.845 INFO:tasks.workunit.client.0.smithi139.stdout: "acting": 2, 2024-09-15T20:32:59.845 INFO:tasks.workunit.client.0.smithi139.stdout: "num_store_stats": 3 2024-09-15T20:32:59.845 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.845 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:32:59.845 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_stats": [ 2024-09-15T20:32:59.845 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.845 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:32:59.845 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 154618822667 2024-09-15T20:32:59.845 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.845 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.845 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:32:59.845 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 42949672986 2024-09-15T20:32:59.845 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.846 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.846 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:32:59.846 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 163208757258 2024-09-15T20:32:59.846 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.846 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:32:59.846 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_by_state": [ 2024-09-15T20:32:59.846 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.846 INFO:tasks.workunit.client.0.smithi139.stdout: "state": "active+undersized+degraded", 2024-09-15T20:32:59.846 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 1 2024-09-15T20:32:59.846 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.846 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:32:59.846 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_by_osd": [ 2024-09-15T20:32:59.846 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.847 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:32:59.847 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 0, 2024-09-15T20:32:59.847 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 1, 2024-09-15T20:32:59.847 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:32:59.847 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.847 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.847 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:32:59.847 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 1, 2024-09-15T20:32:59.847 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 1, 2024-09-15T20:32:59.847 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:32:59.847 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.847 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.847 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:32:59.847 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 0, 2024-09-15T20:32:59.847 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 0, 2024-09-15T20:32:59.848 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:32:59.848 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.848 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:32:59.848 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snaps": [ 2024-09-15T20:32:59.848 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:32:59.848 INFO:tasks.workunit.client.0.smithi139.stdout: "pool": 1, 2024-09-15T20:32:59.848 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snaps": {} 2024-09-15T20:32:59.848 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.848 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:32:59.848 INFO:tasks.workunit.client.0.smithi139.stdout: "servicemap": { 2024-09-15T20:32:59.848 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 5, 2024-09-15T20:32:59.848 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:32:18.523215+0000", 2024-09-15T20:32:59.848 INFO:tasks.workunit.client.0.smithi139.stdout: "services": { 2024-09-15T20:32:59.849 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": { 2024-09-15T20:32:59.849 INFO:tasks.workunit.client.0.smithi139.stdout: "daemons": { 2024-09-15T20:32:59.849 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": "", 2024-09-15T20:32:59.849 INFO:tasks.workunit.client.0.smithi139.stdout: "0": { 2024-09-15T20:32:59.849 INFO:tasks.workunit.client.0.smithi139.stdout: "start_epoch": 0, 2024-09-15T20:32:59.849 INFO:tasks.workunit.client.0.smithi139.stdout: "start_stamp": "0.000000", 2024-09-15T20:32:59.849 INFO:tasks.workunit.client.0.smithi139.stdout: "gid": 0, 2024-09-15T20:32:59.849 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "(unrecognized address family 0)/0", 2024-09-15T20:32:59.849 INFO:tasks.workunit.client.0.smithi139.stdout: "metadata": {}, 2024-09-15T20:32:59.849 INFO:tasks.workunit.client.0.smithi139.stdout: "task_status": {} 2024-09-15T20:32:59.849 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.849 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.849 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.849 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.850 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:32:59.850 INFO:tasks.workunit.client.0.smithi139.stdout: "mgrstat_first_committed": 1, 2024-09-15T20:32:59.850 INFO:tasks.workunit.client.0.smithi139.stdout: "mgrstat_last_committed": 94, 2024-09-15T20:32:59.850 INFO:tasks.workunit.client.0.smithi139.stdout: "logm_first_committed": 1, 2024-09-15T20:32:59.850 INFO:tasks.workunit.client.0.smithi139.stdout: "logm_last_committed": 401, 2024-09-15T20:32:59.850 INFO:tasks.workunit.client.0.smithi139.stdout: "paxos": { 2024-09-15T20:32:59.850 INFO:tasks.workunit.client.0.smithi139.stdout: "first_committed": 1, 2024-09-15T20:32:59.850 INFO:tasks.workunit.client.0.smithi139.stdout: "last_committed": 544, 2024-09-15T20:32:59.850 INFO:tasks.workunit.client.0.smithi139.stdout: "last_pn": 0, 2024-09-15T20:32:59.850 INFO:tasks.workunit.client.0.smithi139.stdout: "accepted_pn": 0 2024-09-15T20:32:59.850 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:32:59.850 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:32:59.851 INFO:tasks.workunit.client.0.smithi139.stderr:report 3235596740 2024-09-15T20:32:59.851 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-15T20:32:59.851 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2024-09-15T20:32:59.851 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2024-09-15T20:32:59.851 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2024-09-15T20:32:59.852 INFO:tasks.workunit.client.0.smithi139.stdout:writing non-divergent object existing_4 2024-09-15T20:32:59.852 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2024-09-15T20:33:00.204 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:33:00.204 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-15T20:32:21.773889+0000 29'100 42:130 [0,2] 0 [0,2] 0 0'0 2024-09-15T20:31:19.998451+0000 0'0 2024-09-15T20:31:19.998451+0000 0 0 periodic scrub scheduled @ 2024-09-16T23:16:44.266472+0000 0 0 2024-09-15T20:33:00.204 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:33:00.204 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:33:00.204 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:33:00.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2024-09-15T20:33:00.329 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T20:33:00.329 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2024-09-15T20:33:00.329 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:33:00.329 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:33:00.329 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:33:00.330 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T20:33:00.330 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:33:00.330 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:33:00.330 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:33:00.330 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:33:00.330 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:33:00.395 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T20:33:00.395 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:33:00.395 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:33:00.395 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:33:00.395 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:33:00.395 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:33:00.854 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:33:00.854 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:33:00.854 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:33:00.854 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:33:00.854 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:33:00.855 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:33:01.031 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822670 2024-09-15T20:33:01.031 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822670 2024-09-15T20:33:01.031 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822670' 2024-09-15T20:33:01.031 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:33:01.032 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:33:01.155 INFO:tasks.workunit.client.0.smithi139.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-15T20:33:01.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-15T20:33:01.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-15T20:33:01.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-15T20:33:01.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:33:01.157 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:33:01.320 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757261 2024-09-15T20:33:01.320 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757261 2024-09-15T20:33:01.320 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822670 2-163208757261' 2024-09-15T20:33:01.320 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:33:01.321 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-154618822670 2024-09-15T20:33:01.321 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:33:01.321 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:33:01.322 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-154618822670 2024-09-15T20:33:01.322 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:33:01.323 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822670 2024-09-15T20:33:01.323 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 154618822670 2024-09-15T20:33:01.323 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 154618822670' 2024-09-15T20:33:01.323 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:33:01.733 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822668 -lt 154618822670 2024-09-15T20:33:01.733 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:33:02.735 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:33:02.735 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:33:03.190 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822670 -lt 154618822670 2024-09-15T20:33:03.191 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:33:03.191 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-163208757261 2024-09-15T20:33:03.191 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:33:03.192 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:33:03.193 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-163208757261 2024-09-15T20:33:03.193 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:33:03.194 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757261 2024-09-15T20:33:03.194 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 163208757261 2024-09-15T20:33:03.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 163208757261' 2024-09-15T20:33:03.195 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:33:03.659 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757261 -lt 163208757261 2024-09-15T20:33:03.659 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:33:03.659 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:33:03.659 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:33:04.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:33:04.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:33:04.225 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:33:04.225 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:33:04.225 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:33:04.226 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:33:04.226 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:33:04.226 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:33:04.629 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:33:04.629 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:33:04.630 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:33:04.630 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:33:05.230 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:33:05.231 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T20:33:05.231 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:33:05.231 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T20:33:05.231 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:33:05.231 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:33:05.331 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:33:05.332 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:33:05.332 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:33:05.333 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:33:05.333 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:33:05.333 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:33:05.333 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:33:05.333 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:33:05.772 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:33:05.773 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:33:05.773 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:33:05.773 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:33:06.322 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:33:06.323 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:33:06.323 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:33:06.323 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:33:06.323 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:33:06.323 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:33:06.323 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:33:06.323 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:33:06.324 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:33:06.890 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:33:06.890 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:33:06.891 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-15T20:33:06.891 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:33:06.891 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:33:07.091 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:33:07.091 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:33:07.092 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:33:07.092 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:33:07.092 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:33:07.092 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:33:07.093 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:33:07.093 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:33:07.517 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:33:07.517 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:33:07.518 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:33:07.518 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:33:08.100 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:33:08.100 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:33:08.100 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:33:08.100 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:33:08.100 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:33:08.100 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:33:08.100 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:33:08.101 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:33:08.101 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:33:08.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:33:08.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:33:08.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-15T20:33:08.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:33:08.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T20:33:09.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:33:09.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:33:09.063 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:33:09.064 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:33:09.064 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:33:09.064 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:33:09.064 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:33:09.064 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:33:09.487 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:33:09.487 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:33:09.488 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:33:09.488 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:33:10.039 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:33:10.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:33:10.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:33:10.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:33:10.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:33:10.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:33:10.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:33:10.040 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:33:10.041 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:33:10.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:33:10.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:33:10.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-15T20:33:10.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:33:10.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-15T20:33:11.383 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:33:11.383 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:33:11.384 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:33:11.384 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:33:11.384 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:33:11.384 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:33:11.384 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:33:11.384 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:33:11.824 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:33:11.824 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:33:11.824 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:33:11.824 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:33:12.380 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:33:12.380 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:33:12.380 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:33:12.380 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:33:12.380 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:33:12.380 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:33:12.380 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:33:12.381 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:33:12.381 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:33:12.929 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:33:12.929 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:33:12.929 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-15T20:33:12.929 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:33:12.929 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-15T20:33:14.530 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:33:14.530 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:33:14.531 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:33:14.531 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:33:14.531 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:33:14.531 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:33:14.531 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:33:14.531 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:33:14.964 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:33:14.964 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:33:14.964 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:33:14.964 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:33:15.546 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:33:15.546 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:33:15.547 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:33:15.547 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:33:15.547 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:33:15.547 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:33:15.547 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:33:15.547 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:33:15.547 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:33:16.133 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:33:16.133 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:33:16.133 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-15T20:33:16.134 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:33:16.134 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-15T20:33:19.335 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:33:19.335 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:33:19.336 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:33:19.336 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:33:19.336 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:33:19.336 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:33:19.336 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:33:19.336 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:33:19.772 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:33:19.772 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:33:19.772 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:33:19.773 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:33:20.337 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:33:20.337 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:33:20.337 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:33:20.338 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:33:20.338 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:33:20.338 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:33:20.338 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:33:20.338 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:33:20.338 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:33:20.901 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:33:20.901 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:33:20.901 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-15T20:33:20.901 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:33:20.901 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-15T20:33:27.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:33:27.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:33:27.304 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:33:27.304 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:33:27.304 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:33:27.304 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:33:27.304 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:33:27.304 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:33:27.751 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:33:27.752 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:33:27.752 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:33:27.752 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:33:28.317 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:33:28.317 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:33:28.317 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:33:28.317 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:33:28.317 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:33:28.317 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:33:28.317 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:33:28.318 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:33:28.318 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:33:28.872 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:33:28.873 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:33:28.873 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-15T20:33:28.873 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:33:28.873 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-15T20:33:36.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:33:36.174 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:33:36.174 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:33:36.174 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:33:36.175 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:33:36.175 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:33:36.175 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:33:36.175 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:33:36.613 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:33:36.613 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:33:36.614 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:33:36.614 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:33:37.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:33:37.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:33:37.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:33:37.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:33:37.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:33:37.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:33:37.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:33:37.225 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:33:37.225 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:33:37.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:33:37.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:33:37.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-15T20:33:37.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-15T20:33:38.328 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:33:38.328 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_fingerprint": "75d6c3f0-55f0-4f53-878f-24c19012eb4d", 2024-09-15T20:33:38.328 INFO:tasks.workunit.client.0.smithi139.stdout: "version": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:33:38.328 INFO:tasks.workunit.client.0.smithi139.stdout: "commit": "fc3e0bff7468fd55478adbae7f75508b749124b6", 2024-09-15T20:33:38.328 INFO:tasks.workunit.client.0.smithi139.stdout: "timestamp": "2024-09-15T20:33:38.328388+0000", 2024-09-15T20:33:38.328 INFO:tasks.workunit.client.0.smithi139.stdout: "tag": "", 2024-09-15T20:33:38.328 INFO:tasks.workunit.client.0.smithi139.stdout: "health": { 2024-09-15T20:33:38.328 INFO:tasks.workunit.client.0.smithi139.stdout: "status": "HEALTH_WARN", 2024-09-15T20:33:38.329 INFO:tasks.workunit.client.0.smithi139.stdout: "checks": { 2024-09-15T20:33:38.329 INFO:tasks.workunit.client.0.smithi139.stdout: "OSDMAP_FLAGS": { 2024-09-15T20:33:38.329 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:33:38.329 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:33:38.329 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-15T20:33:38.329 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 17 2024-09-15T20:33:38.329 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.329 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [], 2024-09-15T20:33:38.329 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:33:38.329 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.329 INFO:tasks.workunit.client.0.smithi139.stdout: "OSD_DOWN": { 2024-09-15T20:33:38.329 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:33:38.329 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:33:38.330 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "1 osds down", 2024-09-15T20:33:38.330 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 1 2024-09-15T20:33:38.330 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.330 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:33:38.330 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.330 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "osd.1 (root=default,host=smithi139) is down" 2024-09-15T20:33:38.330 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.330 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:33:38.330 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:33:38.330 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.330 INFO:tasks.workunit.client.0.smithi139.stdout: "PG_DEGRADED": { 2024-09-15T20:33:38.330 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:33:38.330 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:33:38.330 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2024-09-15T20:33:38.331 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 2 2024-09-15T20:33:38.331 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.331 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:33:38.331 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.331 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.0 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T20:33:38.331 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.331 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:33:38.331 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:33:38.331 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.331 INFO:tasks.workunit.client.0.smithi139.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-15T20:33:38.331 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:33:38.331 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:33:38.331 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-15T20:33:38.332 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 1 2024-09-15T20:33:38.332 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.332 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:33:38.332 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.332 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "application not enabled on pool 'test'" 2024-09-15T20:33:38.332 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.332 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.332 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-15T20:33:38.332 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.332 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:33:38.332 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:33:38.332 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.332 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.333 INFO:tasks.workunit.client.0.smithi139.stdout: "mutes": [] 2024-09-15T20:33:38.333 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.333 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap_first_committed": 1, 2024-09-15T20:33:38.333 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap_last_committed": 1, 2024-09-15T20:33:38.333 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap": { 2024-09-15T20:33:38.333 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 1, 2024-09-15T20:33:38.333 INFO:tasks.workunit.client.0.smithi139.stdout: "fsid": "0e220df1-ac9c-433e-8885-cf9f7aab293e", 2024-09-15T20:33:38.333 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:29:28.018156Z", 2024-09-15T20:33:38.333 INFO:tasks.workunit.client.0.smithi139.stdout: "created": "2024-09-15T20:29:28.018156Z", 2024-09-15T20:33:38.333 INFO:tasks.workunit.client.0.smithi139.stdout: "min_mon_release": 19, 2024-09-15T20:33:38.333 INFO:tasks.workunit.client.0.smithi139.stdout: "min_mon_release_name": "squid", 2024-09-15T20:33:38.333 INFO:tasks.workunit.client.0.smithi139.stdout: "election_strategy": 1, 2024-09-15T20:33:38.333 INFO:tasks.workunit.client.0.smithi139.stdout: "disallowed_leaders: ": "", 2024-09-15T20:33:38.333 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode": false, 2024-09-15T20:33:38.334 INFO:tasks.workunit.client.0.smithi139.stdout: "tiebreaker_mon": "", 2024-09-15T20:33:38.334 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_ranks: ": "", 2024-09-15T20:33:38.334 INFO:tasks.workunit.client.0.smithi139.stdout: "features": { 2024-09-15T20:33:38.334 INFO:tasks.workunit.client.0.smithi139.stdout: "persistent": [ 2024-09-15T20:33:38.334 INFO:tasks.workunit.client.0.smithi139.stdout: "kraken", 2024-09-15T20:33:38.334 INFO:tasks.workunit.client.0.smithi139.stdout: "luminous", 2024-09-15T20:33:38.334 INFO:tasks.workunit.client.0.smithi139.stdout: "mimic", 2024-09-15T20:33:38.334 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap-prune", 2024-09-15T20:33:38.334 INFO:tasks.workunit.client.0.smithi139.stdout: "nautilus", 2024-09-15T20:33:38.334 INFO:tasks.workunit.client.0.smithi139.stdout: "octopus", 2024-09-15T20:33:38.334 INFO:tasks.workunit.client.0.smithi139.stdout: "pacific", 2024-09-15T20:33:38.334 INFO:tasks.workunit.client.0.smithi139.stdout: "elector-pinging", 2024-09-15T20:33:38.334 INFO:tasks.workunit.client.0.smithi139.stdout: "quincy", 2024-09-15T20:33:38.334 INFO:tasks.workunit.client.0.smithi139.stdout: "reef", 2024-09-15T20:33:38.335 INFO:tasks.workunit.client.0.smithi139.stdout: "squid" 2024-09-15T20:33:38.335 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:33:38.335 INFO:tasks.workunit.client.0.smithi139.stdout: "optional": [] 2024-09-15T20:33:38.335 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.335 INFO:tasks.workunit.client.0.smithi139.stdout: "mons": [ 2024-09-15T20:33:38.335 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.335 INFO:tasks.workunit.client.0.smithi139.stdout: "rank": 0, 2024-09-15T20:33:38.335 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "a", 2024-09-15T20:33:38.335 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:33:38.335 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:33:38.336 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.336 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:33:38.336 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:7115", 2024-09-15T20:33:38.336 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 0 2024-09-15T20:33:38.336 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.336 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.336 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.336 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:7115/0", 2024-09-15T20:33:38.336 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-15T20:33:38.336 INFO:tasks.workunit.client.0.smithi139.stdout: "priority": 0, 2024-09-15T20:33:38.336 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 0, 2024-09-15T20:33:38.336 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_location": "{}" 2024-09-15T20:33:38.336 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.337 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.337 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.337 INFO:tasks.workunit.client.0.smithi139.stdout: "quorum": [ 2024-09-15T20:33:38.337 INFO:tasks.workunit.client.0.smithi139.stdout: 0 2024-09-15T20:33:38.337 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:33:38.337 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap": { 2024-09-15T20:33:38.337 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 42, 2024-09-15T20:33:38.337 INFO:tasks.workunit.client.0.smithi139.stdout: "fsid": "0e220df1-ac9c-433e-8885-cf9f7aab293e", 2024-09-15T20:33:38.337 INFO:tasks.workunit.client.0.smithi139.stdout: "created": "2024-09-15T20:29:28.963686+0000", 2024-09-15T20:33:38.337 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:32:21.403941+0000", 2024-09-15T20:33:38.337 INFO:tasks.workunit.client.0.smithi139.stdout: "last_up_change": "2024-09-15T20:32:19.646667+0000", 2024-09-15T20:33:38.337 INFO:tasks.workunit.client.0.smithi139.stdout: "last_in_change": "2024-09-15T20:32:21.135530+0000", 2024-09-15T20:33:38.337 INFO:tasks.workunit.client.0.smithi139.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T20:33:38.337 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_num": 5800384, 2024-09-15T20:33:38.338 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_set": [ 2024-09-15T20:33:38.338 INFO:tasks.workunit.client.0.smithi139.stdout: "nodown", 2024-09-15T20:33:38.338 INFO:tasks.workunit.client.0.smithi139.stdout: "noin", 2024-09-15T20:33:38.338 INFO:tasks.workunit.client.0.smithi139.stdout: "noout", 2024-09-15T20:33:38.338 INFO:tasks.workunit.client.0.smithi139.stdout: "pglog_hardlimit", 2024-09-15T20:33:38.338 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snapdirs", 2024-09-15T20:33:38.338 INFO:tasks.workunit.client.0.smithi139.stdout: "recovery_deletes", 2024-09-15T20:33:38.338 INFO:tasks.workunit.client.0.smithi139.stdout: "sortbitwise" 2024-09-15T20:33:38.338 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:33:38.338 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_version": 9, 2024-09-15T20:33:38.338 INFO:tasks.workunit.client.0.smithi139.stdout: "full_ratio": 0.99000000953674316, 2024-09-15T20:33:38.338 INFO:tasks.workunit.client.0.smithi139.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-15T20:33:38.338 INFO:tasks.workunit.client.0.smithi139.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T20:33:38.338 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_snapshot": "", 2024-09-15T20:33:38.339 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_max": 1, 2024-09-15T20:33:38.339 INFO:tasks.workunit.client.0.smithi139.stdout: "max_osd": 3, 2024-09-15T20:33:38.339 INFO:tasks.workunit.client.0.smithi139.stdout: "require_min_compat_client": "luminous", 2024-09-15T20:33:38.339 INFO:tasks.workunit.client.0.smithi139.stdout: "min_compat_client": "jewel", 2024-09-15T20:33:38.339 INFO:tasks.workunit.client.0.smithi139.stdout: "require_osd_release": "squid", 2024-09-15T20:33:38.339 INFO:tasks.workunit.client.0.smithi139.stdout: "allow_crimson": false, 2024-09-15T20:33:38.339 INFO:tasks.workunit.client.0.smithi139.stdout: "pools": [ 2024-09-15T20:33:38.339 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.339 INFO:tasks.workunit.client.0.smithi139.stdout: "pool": 1, 2024-09-15T20:33:38.339 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_name": "test", 2024-09-15T20:33:38.339 INFO:tasks.workunit.client.0.smithi139.stdout: "create_time": "2024-09-15T20:31:19.939932+0000", 2024-09-15T20:33:38.339 INFO:tasks.workunit.client.0.smithi139.stdout: "flags": 1, 2024-09-15T20:33:38.339 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_names": "hashpspool", 2024-09-15T20:33:38.339 INFO:tasks.workunit.client.0.smithi139.stdout: "type": 1, 2024-09-15T20:33:38.340 INFO:tasks.workunit.client.0.smithi139.stdout: "size": 3, 2024-09-15T20:33:38.340 INFO:tasks.workunit.client.0.smithi139.stdout: "min_size": 2, 2024-09-15T20:33:38.340 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_rule": 0, 2024-09-15T20:33:38.340 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_count": 0, 2024-09-15T20:33:38.340 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_target": 0, 2024-09-15T20:33:38.340 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T20:33:38.340 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T20:33:38.340 INFO:tasks.workunit.client.0.smithi139.stdout: "is_stretch_pool": false, 2024-09-15T20:33:38.340 INFO:tasks.workunit.client.0.smithi139.stdout: "object_hash": 2, 2024-09-15T20:33:38.340 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_autoscale_mode": "off", 2024-09-15T20:33:38.340 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num": 1, 2024-09-15T20:33:38.340 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_placement_num": 1, 2024-09-15T20:33:38.340 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_placement_num_target": 1, 2024-09-15T20:33:38.341 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num_target": 1, 2024-09-15T20:33:38.341 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num_pending": 1, 2024-09-15T20:33:38.341 INFO:tasks.workunit.client.0.smithi139.stdout: "last_pg_merge_meta": { 2024-09-15T20:33:38.341 INFO:tasks.workunit.client.0.smithi139.stdout: "source_pgid": "0.0", 2024-09-15T20:33:38.341 INFO:tasks.workunit.client.0.smithi139.stdout: "ready_epoch": 0, 2024-09-15T20:33:38.341 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_started": 0, 2024-09-15T20:33:38.341 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_clean": 0, 2024-09-15T20:33:38.341 INFO:tasks.workunit.client.0.smithi139.stdout: "source_version": "0'0", 2024-09-15T20:33:38.341 INFO:tasks.workunit.client.0.smithi139.stdout: "target_version": "0'0" 2024-09-15T20:33:38.341 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.341 INFO:tasks.workunit.client.0.smithi139.stdout: "last_change": "29", 2024-09-15T20:33:38.341 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend": "0", 2024-09-15T20:33:38.341 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T20:33:38.341 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T20:33:38.342 INFO:tasks.workunit.client.0.smithi139.stdout: "auid": 0, 2024-09-15T20:33:38.342 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_mode": "selfmanaged", 2024-09-15T20:33:38.342 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_seq": 0, 2024-09-15T20:33:38.342 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_epoch": 0, 2024-09-15T20:33:38.342 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_snaps": [], 2024-09-15T20:33:38.342 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_snaps": "[]", 2024-09-15T20:33:38.342 INFO:tasks.workunit.client.0.smithi139.stdout: "quota_max_bytes": 0, 2024-09-15T20:33:38.342 INFO:tasks.workunit.client.0.smithi139.stdout: "quota_max_objects": 0, 2024-09-15T20:33:38.342 INFO:tasks.workunit.client.0.smithi139.stdout: "tiers": [], 2024-09-15T20:33:38.342 INFO:tasks.workunit.client.0.smithi139.stdout: "tier_of": -1, 2024-09-15T20:33:38.342 INFO:tasks.workunit.client.0.smithi139.stdout: "read_tier": -1, 2024-09-15T20:33:38.342 INFO:tasks.workunit.client.0.smithi139.stdout: "write_tier": -1, 2024-09-15T20:33:38.342 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_mode": "none", 2024-09-15T20:33:38.342 INFO:tasks.workunit.client.0.smithi139.stdout: "target_max_bytes": 0, 2024-09-15T20:33:38.342 INFO:tasks.workunit.client.0.smithi139.stdout: "target_max_objects": 0, 2024-09-15T20:33:38.343 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T20:33:38.343 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T20:33:38.343 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T20:33:38.343 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_min_flush_age": 0, 2024-09-15T20:33:38.343 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_min_evict_age": 0, 2024-09-15T20:33:38.343 INFO:tasks.workunit.client.0.smithi139.stdout: "erasure_code_profile": "", 2024-09-15T20:33:38.343 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_params": { 2024-09-15T20:33:38.343 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "none" 2024-09-15T20:33:38.343 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.343 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_period": 0, 2024-09-15T20:33:38.343 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_count": 0, 2024-09-15T20:33:38.343 INFO:tasks.workunit.client.0.smithi139.stdout: "use_gmt_hitset": true, 2024-09-15T20:33:38.343 INFO:tasks.workunit.client.0.smithi139.stdout: "min_read_recency_for_promote": 0, 2024-09-15T20:33:38.343 INFO:tasks.workunit.client.0.smithi139.stdout: "min_write_recency_for_promote": 0, 2024-09-15T20:33:38.344 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T20:33:38.344 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_search_last_n": 0, 2024-09-15T20:33:38.344 INFO:tasks.workunit.client.0.smithi139.stdout: "grade_table": [], 2024-09-15T20:33:38.344 INFO:tasks.workunit.client.0.smithi139.stdout: "stripe_width": 0, 2024-09-15T20:33:38.344 INFO:tasks.workunit.client.0.smithi139.stdout: "expected_num_objects": 0, 2024-09-15T20:33:38.344 INFO:tasks.workunit.client.0.smithi139.stdout: "fast_read": false, 2024-09-15T20:33:38.344 INFO:tasks.workunit.client.0.smithi139.stdout: "options": {}, 2024-09-15T20:33:38.344 INFO:tasks.workunit.client.0.smithi139.stdout: "application_metadata": {}, 2024-09-15T20:33:38.344 INFO:tasks.workunit.client.0.smithi139.stdout: "read_balance": { 2024-09-15T20:33:38.344 INFO:tasks.workunit.client.0.smithi139.stdout: "score_type": "Fair distribution", 2024-09-15T20:33:38.344 INFO:tasks.workunit.client.0.smithi139.stdout: "score_acting": 2.9900000095367432, 2024-09-15T20:33:38.344 INFO:tasks.workunit.client.0.smithi139.stdout: "score_stable": 2.9900000095367432, 2024-09-15T20:33:38.344 INFO:tasks.workunit.client.0.smithi139.stdout: "optimal_score": 0.67000001668930054, 2024-09-15T20:33:38.344 INFO:tasks.workunit.client.0.smithi139.stdout: "raw_score_acting": 2, 2024-09-15T20:33:38.345 INFO:tasks.workunit.client.0.smithi139.stdout: "raw_score_stable": 2, 2024-09-15T20:33:38.345 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity_weighted": 1, 2024-09-15T20:33:38.345 INFO:tasks.workunit.client.0.smithi139.stdout: "average_primary_affinity": 1, 2024-09-15T20:33:38.345 INFO:tasks.workunit.client.0.smithi139.stdout: "average_primary_affinity_weighted": 1 2024-09-15T20:33:38.345 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.345 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.345 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:33:38.345 INFO:tasks.workunit.client.0.smithi139.stdout: "osds": [ 2024-09-15T20:33:38.345 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.345 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:33:38.345 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "457b5e59-ab0b-4ff7-8ba7-80b5ae6b10f2", 2024-09-15T20:33:38.345 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 1, 2024-09-15T20:33:38.345 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:33:38.345 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:33:38.346 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:33:38.346 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 5, 2024-09-15T20:33:38.346 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 29, 2024-09-15T20:33:38.346 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 36, 2024-09-15T20:33:38.346 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 41, 2024-09-15T20:33:38.346 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 30, 2024-09-15T20:33:38.346 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:33:38.346 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:33:38.346 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:33:38.346 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.346 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:33:38.346 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6802", 2024-09-15T20:33:38.346 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1803718560 2024-09-15T20:33:38.346 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.347 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.347 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:33:38.347 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6803", 2024-09-15T20:33:38.347 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1803718560 2024-09-15T20:33:38.347 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.347 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.347 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.347 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:33:38.347 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:33:38.347 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.347 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:33:38.347 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6804", 2024-09-15T20:33:38.347 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1803718560 2024-09-15T20:33:38.347 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.347 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.348 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:33:38.348 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6805", 2024-09-15T20:33:38.348 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1803718560 2024-09-15T20:33:38.348 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.348 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.348 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.348 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:33:38.348 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:33:38.348 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.348 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:33:38.348 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6808", 2024-09-15T20:33:38.348 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1803718560 2024-09-15T20:33:38.348 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.349 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.349 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:33:38.349 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6809", 2024-09-15T20:33:38.349 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1803718560 2024-09-15T20:33:38.349 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.349 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.349 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.349 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:33:38.349 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:33:38.349 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.349 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:33:38.349 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6806", 2024-09-15T20:33:38.349 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1803718560 2024-09-15T20:33:38.349 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.350 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.350 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:33:38.350 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6807", 2024-09-15T20:33:38.350 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1803718560 2024-09-15T20:33:38.350 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.350 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.350 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.350 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6803/1803718560", 2024-09-15T20:33:38.350 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6805/1803718560", 2024-09-15T20:33:38.350 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1803718560", 2024-09-15T20:33:38.350 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1803718560", 2024-09-15T20:33:38.350 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:33:38.350 INFO:tasks.workunit.client.0.smithi139.stdout: "exists", 2024-09-15T20:33:38.350 INFO:tasks.workunit.client.0.smithi139.stdout: "up" 2024-09-15T20:33:38.351 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.351 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.351 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.351 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:33:38.351 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "037d24c1-96ec-4453-b5af-a66b6de4e720", 2024-09-15T20:33:38.351 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 0, 2024-09-15T20:33:38.351 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:33:38.351 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:33:38.351 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:33:38.351 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 0, 2024-09-15T20:33:38.351 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 0, 2024-09-15T20:33:38.351 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 10, 2024-09-15T20:33:38.351 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 23, 2024-09-15T20:33:38.351 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 31, 2024-09-15T20:33:38.352 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:33:38.352 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:33:38.352 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:33:38.352 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.352 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:33:38.352 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6810", 2024-09-15T20:33:38.352 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 4110363107 2024-09-15T20:33:38.352 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.352 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.352 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:33:38.352 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6811", 2024-09-15T20:33:38.352 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 4110363107 2024-09-15T20:33:38.352 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.352 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.353 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.353 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:33:38.353 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:33:38.353 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.353 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:33:38.353 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6812", 2024-09-15T20:33:38.353 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 4110363107 2024-09-15T20:33:38.353 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.353 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.353 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:33:38.353 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6813", 2024-09-15T20:33:38.353 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 4110363107 2024-09-15T20:33:38.353 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.353 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.354 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.354 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:33:38.354 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:33:38.354 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.354 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:33:38.354 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6816", 2024-09-15T20:33:38.354 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 4110363107 2024-09-15T20:33:38.354 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.354 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.354 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:33:38.354 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6817", 2024-09-15T20:33:38.354 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 4110363107 2024-09-15T20:33:38.354 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.354 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.355 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.355 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:33:38.355 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:33:38.355 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.355 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:33:38.355 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6814", 2024-09-15T20:33:38.355 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 4110363107 2024-09-15T20:33:38.355 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.355 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.355 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:33:38.355 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6815", 2024-09-15T20:33:38.355 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 4110363107 2024-09-15T20:33:38.355 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.355 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.356 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.356 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6811/4110363107", 2024-09-15T20:33:38.356 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6813/4110363107", 2024-09-15T20:33:38.356 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6817/4110363107", 2024-09-15T20:33:38.356 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6815/4110363107", 2024-09-15T20:33:38.356 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:33:38.356 INFO:tasks.workunit.client.0.smithi139.stdout: "exists" 2024-09-15T20:33:38.356 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.356 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.356 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.356 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:33:38.356 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "24e3e077-bb9a-4ae3-9be6-20ce6b775ae4", 2024-09-15T20:33:38.356 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 1, 2024-09-15T20:33:38.356 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:33:38.357 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:33:38.357 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:33:38.357 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 15, 2024-09-15T20:33:38.357 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 29, 2024-09-15T20:33:38.357 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 38, 2024-09-15T20:33:38.357 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 0, 2024-09-15T20:33:38.357 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 32, 2024-09-15T20:33:38.357 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:33:38.357 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:33:38.357 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:33:38.357 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.357 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:33:38.357 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6810", 2024-09-15T20:33:38.357 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3470466229 2024-09-15T20:33:38.358 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.358 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.358 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:33:38.358 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6811", 2024-09-15T20:33:38.358 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3470466229 2024-09-15T20:33:38.358 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.358 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.358 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.358 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:33:38.358 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:33:38.358 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.358 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:33:38.358 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6812", 2024-09-15T20:33:38.358 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3470466229 2024-09-15T20:33:38.359 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.359 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.359 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:33:38.359 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6813", 2024-09-15T20:33:38.359 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3470466229 2024-09-15T20:33:38.359 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.359 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.359 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.359 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:33:38.359 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:33:38.359 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.359 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:33:38.359 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6816", 2024-09-15T20:33:38.359 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3470466229 2024-09-15T20:33:38.359 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.360 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.360 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:33:38.360 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6817", 2024-09-15T20:33:38.360 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3470466229 2024-09-15T20:33:38.360 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.360 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.360 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.360 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:33:38.360 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:33:38.360 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.360 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:33:38.360 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6814", 2024-09-15T20:33:38.360 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3470466229 2024-09-15T20:33:38.361 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.361 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.361 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:33:38.361 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6815", 2024-09-15T20:33:38.361 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3470466229 2024-09-15T20:33:38.361 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.361 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.361 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.361 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6811/3470466229", 2024-09-15T20:33:38.361 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6813/3470466229", 2024-09-15T20:33:38.361 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3470466229", 2024-09-15T20:33:38.361 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3470466229", 2024-09-15T20:33:38.361 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:33:38.361 INFO:tasks.workunit.client.0.smithi139.stdout: "exists", 2024-09-15T20:33:38.362 INFO:tasks.workunit.client.0.smithi139.stdout: "up" 2024-09-15T20:33:38.362 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.362 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.362 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:33:38.362 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_xinfo": [ 2024-09-15T20:33:38.362 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.362 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:33:38.362 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:32:06.380843+0000", 2024-09-15T20:33:38.362 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:33:38.362 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:33:38.362 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:33:38.362 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:33:38.362 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:30:04.676305+0000", 2024-09-15T20:33:38.363 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:33:38.363 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.363 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.363 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:33:38.363 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:32:06.958649+0000", 2024-09-15T20:33:38.363 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:33:38.363 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:33:38.363 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:33:38.363 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:33:38.363 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:30:40.276160+0000", 2024-09-15T20:33:38.363 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:33:38.363 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.363 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.363 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:33:38.363 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:32:07.595692+0000", 2024-09-15T20:33:38.364 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:33:38.364 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:33:38.364 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:33:38.364 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:33:38.364 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:31:15.824964+0000", 2024-09-15T20:33:38.364 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:33:38.364 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.364 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:33:38.364 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap": [], 2024-09-15T20:33:38.364 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap_items": [], 2024-09-15T20:33:38.364 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap_primaries": [], 2024-09-15T20:33:38.364 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_temp": [], 2024-09-15T20:33:38.364 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_temp": [], 2024-09-15T20:33:38.364 INFO:tasks.workunit.client.0.smithi139.stdout: "blocklist": {}, 2024-09-15T20:33:38.365 INFO:tasks.workunit.client.0.smithi139.stdout: "range_blocklist": {}, 2024-09-15T20:33:38.365 INFO:tasks.workunit.client.0.smithi139.stdout: "erasure_code_profiles": { 2024-09-15T20:33:38.365 INFO:tasks.workunit.client.0.smithi139.stdout: "default": { 2024-09-15T20:33:38.365 INFO:tasks.workunit.client.0.smithi139.stdout: "k": "2", 2024-09-15T20:33:38.365 INFO:tasks.workunit.client.0.smithi139.stdout: "m": "2", 2024-09-15T20:33:38.365 INFO:tasks.workunit.client.0.smithi139.stdout: "plugin": "jerasure", 2024-09-15T20:33:38.365 INFO:tasks.workunit.client.0.smithi139.stdout: "technique": "reed_sol_van" 2024-09-15T20:33:38.365 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.365 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.365 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_snaps_queue": [], 2024-09-15T20:33:38.365 INFO:tasks.workunit.client.0.smithi139.stdout: "new_removed_snaps": [], 2024-09-15T20:33:38.365 INFO:tasks.workunit.client.0.smithi139.stdout: "new_purged_snaps": [], 2024-09-15T20:33:38.365 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_node_flags": {}, 2024-09-15T20:33:38.365 INFO:tasks.workunit.client.0.smithi139.stdout: "device_class_flags": {}, 2024-09-15T20:33:38.366 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode": { 2024-09-15T20:33:38.366 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode_enabled": false, 2024-09-15T20:33:38.366 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_bucket_count": 0, 2024-09-15T20:33:38.366 INFO:tasks.workunit.client.0.smithi139.stdout: "degraded_stretch_mode": 0, 2024-09-15T20:33:38.366 INFO:tasks.workunit.client.0.smithi139.stdout: "recovering_stretch_mode": 0, 2024-09-15T20:33:38.366 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode_bucket": 0 2024-09-15T20:33:38.366 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.366 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.366 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_metadata": [ 2024-09-15T20:33:38.366 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.366 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:33:38.366 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:33:38.366 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6804/1803718560,v1:127.0.0.1:6805/1803718560]", 2024-09-15T20:33:38.366 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:33:38.367 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:33:38.367 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:33:38.367 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:33:38.367 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:33:38.367 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:33:38.367 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:33:38.367 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:33:38.367 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:33:38.367 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:33:38.367 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:33:38.367 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-15T20:33:38.367 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:33:38.367 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:33:38.367 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:33:38.367 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:33:38.368 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:33:38.368 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:33:38.368 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:33:38.368 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:33:38.368 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:33:38.368 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:33:38.368 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:29:35.554331Z", 2024-09-15T20:33:38.368 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:33:38.368 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:33:38.368 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:33:38.368 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:33:38.368 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:33:38.368 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:33:38.368 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:33:38.369 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6802/1803718560,v1:127.0.0.1:6803/1803718560]", 2024-09-15T20:33:38.369 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:33:38.369 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1803718560,v1:127.0.0.1:6809/1803718560]", 2024-09-15T20:33:38.369 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1803718560,v1:127.0.0.1:6807/1803718560]", 2024-09-15T20:33:38.369 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:33:38.369 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:33:38.369 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:33:38.369 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:33:38.369 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:33:38.369 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:33:38.369 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:33:38.369 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:33:38.369 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:33:38.369 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/0", 2024-09-15T20:33:38.370 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:33:38.370 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:33:38.370 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:33:38.370 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.370 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.370 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:33:38.370 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:33:38.370 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6812/4110363107,v1:127.0.0.1:6813/4110363107]", 2024-09-15T20:33:38.370 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:33:38.370 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:33:38.370 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:33:38.370 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:33:38.370 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:33:38.370 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:33:38.371 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:33:38.371 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:33:38.371 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:33:38.371 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:33:38.371 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:33:38.371 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-15T20:33:38.371 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:33:38.371 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:33:38.371 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:33:38.371 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:33:38.371 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:33:38.371 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:33:38.371 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:33:38.371 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:33:38.372 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:33:38.372 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:33:38.372 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:30:12.116512Z", 2024-09-15T20:33:38.372 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:33:38.372 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:33:38.372 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:33:38.372 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:33:38.372 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:33:38.372 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:33:38.372 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:33:38.372 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6810/4110363107,v1:127.0.0.1:6811/4110363107]", 2024-09-15T20:33:38.372 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:33:38.372 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/4110363107,v1:127.0.0.1:6817/4110363107]", 2024-09-15T20:33:38.372 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/4110363107,v1:127.0.0.1:6815/4110363107]", 2024-09-15T20:33:38.373 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:33:38.373 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:33:38.373 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:33:38.373 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:33:38.373 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:33:38.373 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:33:38.373 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:33:38.373 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:33:38.373 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:33:38.373 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/1", 2024-09-15T20:33:38.373 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:33:38.373 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:33:38.373 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:33:38.373 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.373 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.374 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:33:38.374 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:33:38.374 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6812/3470466229,v1:127.0.0.1:6813/3470466229]", 2024-09-15T20:33:38.374 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:33:38.374 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:33:38.374 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:33:38.374 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:33:38.374 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:33:38.374 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:33:38.374 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:33:38.374 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:33:38.374 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:33:38.374 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:33:38.374 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:33:38.375 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-15T20:33:38.375 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:33:38.375 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:33:38.375 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:33:38.375 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:33:38.375 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:33:38.375 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:33:38.375 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:33:38.375 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:33:38.375 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:33:38.375 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:33:38.375 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:30:47.443148Z", 2024-09-15T20:33:38.375 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:33:38.375 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:33:38.376 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:33:38.376 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:33:38.376 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:33:38.376 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:33:38.376 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:33:38.376 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6810/3470466229,v1:127.0.0.1:6811/3470466229]", 2024-09-15T20:33:38.376 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:33:38.376 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3470466229,v1:127.0.0.1:6817/3470466229]", 2024-09-15T20:33:38.376 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3470466229,v1:127.0.0.1:6815/3470466229]", 2024-09-15T20:33:38.376 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:33:38.376 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:33:38.376 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:33:38.376 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:33:38.376 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:33:38.377 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:33:38.377 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:33:38.377 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:33:38.377 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:33:38.377 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/2", 2024-09-15T20:33:38.377 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:33:38.377 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:33:38.377 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:33:38.377 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.377 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:33:38.377 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_clean_epochs": { 2024-09-15T20:33:38.377 INFO:tasks.workunit.client.0.smithi139.stdout: "min_last_epoch_clean": 0, 2024-09-15T20:33:38.377 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_clean": { 2024-09-15T20:33:38.377 INFO:tasks.workunit.client.0.smithi139.stdout: "per_pool": [] 2024-09-15T20:33:38.378 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.378 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_epochs": [ 2024-09-15T20:33:38.378 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.378 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:33:38.378 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 36 2024-09-15T20:33:38.378 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.378 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.378 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:33:38.378 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 38 2024-09-15T20:33:38.378 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.378 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.378 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.378 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_first_committed": 1, 2024-09-15T20:33:38.378 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_last_committed": 42, 2024-09-15T20:33:38.379 INFO:tasks.workunit.client.0.smithi139.stdout: "crushmap": { 2024-09-15T20:33:38.379 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": [ 2024-09-15T20:33:38.379 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.379 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:33:38.379 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.0", 2024-09-15T20:33:38.379 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:33:38.379 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.379 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.379 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:33:38.379 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.1", 2024-09-15T20:33:38.379 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:33:38.379 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.379 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.379 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:33:38.380 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.2", 2024-09-15T20:33:38.380 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:33:38.380 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.380 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:33:38.380 INFO:tasks.workunit.client.0.smithi139.stdout: "types": [ 2024-09-15T20:33:38.380 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.380 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 0, 2024-09-15T20:33:38.380 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd" 2024-09-15T20:33:38.380 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.380 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.380 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:33:38.380 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "host" 2024-09-15T20:33:38.380 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.381 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.381 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 2, 2024-09-15T20:33:38.381 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "chassis" 2024-09-15T20:33:38.381 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.381 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.381 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 3, 2024-09-15T20:33:38.381 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "rack" 2024-09-15T20:33:38.381 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.381 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.381 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 4, 2024-09-15T20:33:38.381 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "row" 2024-09-15T20:33:38.381 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.381 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.381 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 5, 2024-09-15T20:33:38.382 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "pdu" 2024-09-15T20:33:38.382 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.382 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.382 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 6, 2024-09-15T20:33:38.382 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "pod" 2024-09-15T20:33:38.382 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.382 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.382 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 7, 2024-09-15T20:33:38.382 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "room" 2024-09-15T20:33:38.382 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.382 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.382 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 8, 2024-09-15T20:33:38.382 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "datacenter" 2024-09-15T20:33:38.382 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.382 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.383 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 9, 2024-09-15T20:33:38.383 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "zone" 2024-09-15T20:33:38.383 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.383 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.383 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 10, 2024-09-15T20:33:38.383 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "region" 2024-09-15T20:33:38.383 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.383 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.383 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:33:38.383 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "root" 2024-09-15T20:33:38.383 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.383 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:33:38.383 INFO:tasks.workunit.client.0.smithi139.stdout: "buckets": [ 2024-09-15T20:33:38.384 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.384 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -1, 2024-09-15T20:33:38.384 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "default", 2024-09-15T20:33:38.384 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:33:38.384 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "root", 2024-09-15T20:33:38.384 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:33:38.384 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:33:38.384 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:33:38.384 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:33:38.384 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.384 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -3, 2024-09-15T20:33:38.384 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:33:38.384 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:33:38.384 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.385 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.385 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.385 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.385 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -2, 2024-09-15T20:33:38.385 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "default~hdd", 2024-09-15T20:33:38.385 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:33:38.385 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "root", 2024-09-15T20:33:38.385 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:33:38.385 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:33:38.385 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:33:38.385 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:33:38.385 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.385 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -4, 2024-09-15T20:33:38.386 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:33:38.386 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:33:38.386 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.386 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.386 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.386 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.386 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -3, 2024-09-15T20:33:38.386 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "smithi139", 2024-09-15T20:33:38.386 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:33:38.386 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "host", 2024-09-15T20:33:38.386 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:33:38.386 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:33:38.386 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:33:38.386 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:33:38.386 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.387 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:33:38.387 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:33:38.387 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:33:38.387 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.387 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.387 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:33:38.387 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:33:38.387 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 1 2024-09-15T20:33:38.387 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.387 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.387 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:33:38.387 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:33:38.387 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 2 2024-09-15T20:33:38.387 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.388 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.388 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.388 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.388 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -4, 2024-09-15T20:33:38.388 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "smithi139~hdd", 2024-09-15T20:33:38.388 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:33:38.388 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "host", 2024-09-15T20:33:38.388 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:33:38.388 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:33:38.388 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:33:38.388 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:33:38.388 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.388 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:33:38.388 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:33:38.389 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:33:38.389 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.389 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.389 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:33:38.389 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:33:38.389 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 1 2024-09-15T20:33:38.389 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.389 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.389 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:33:38.389 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:33:38.389 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 2 2024-09-15T20:33:38.389 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.389 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.389 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.390 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:33:38.390 INFO:tasks.workunit.client.0.smithi139.stdout: "rules": [ 2024-09-15T20:33:38.390 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.390 INFO:tasks.workunit.client.0.smithi139.stdout: "rule_id": 0, 2024-09-15T20:33:38.390 INFO:tasks.workunit.client.0.smithi139.stdout: "rule_name": "replicated_rule", 2024-09-15T20:33:38.390 INFO:tasks.workunit.client.0.smithi139.stdout: "type": 1, 2024-09-15T20:33:38.390 INFO:tasks.workunit.client.0.smithi139.stdout: "steps": [ 2024-09-15T20:33:38.390 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.390 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "take", 2024-09-15T20:33:38.390 INFO:tasks.workunit.client.0.smithi139.stdout: "item": -1, 2024-09-15T20:33:38.390 INFO:tasks.workunit.client.0.smithi139.stdout: "item_name": "default" 2024-09-15T20:33:38.390 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.390 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.391 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "choose_firstn", 2024-09-15T20:33:38.391 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 0, 2024-09-15T20:33:38.391 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "osd" 2024-09-15T20:33:38.391 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.391 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.391 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "emit" 2024-09-15T20:33:38.391 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.391 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:33:38.391 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.391 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:33:38.391 INFO:tasks.workunit.client.0.smithi139.stdout: "tunables": { 2024-09-15T20:33:38.391 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_local_tries": 0, 2024-09-15T20:33:38.391 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_local_fallback_tries": 0, 2024-09-15T20:33:38.391 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_total_tries": 50, 2024-09-15T20:33:38.392 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_descend_once": 1, 2024-09-15T20:33:38.392 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_vary_r": 1, 2024-09-15T20:33:38.392 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_stable": 1, 2024-09-15T20:33:38.392 INFO:tasks.workunit.client.0.smithi139.stdout: "msr_descents": 100, 2024-09-15T20:33:38.392 INFO:tasks.workunit.client.0.smithi139.stdout: "msr_collision_tries": 100, 2024-09-15T20:33:38.392 INFO:tasks.workunit.client.0.smithi139.stdout: "straw_calc_version": 1, 2024-09-15T20:33:38.392 INFO:tasks.workunit.client.0.smithi139.stdout: "allowed_bucket_algs": 54, 2024-09-15T20:33:38.392 INFO:tasks.workunit.client.0.smithi139.stdout: "profile": "jewel", 2024-09-15T20:33:38.392 INFO:tasks.workunit.client.0.smithi139.stdout: "optimal_tunables": 1, 2024-09-15T20:33:38.392 INFO:tasks.workunit.client.0.smithi139.stdout: "legacy_tunables": 0, 2024-09-15T20:33:38.392 INFO:tasks.workunit.client.0.smithi139.stdout: "minimum_required_version": "jewel", 2024-09-15T20:33:38.392 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables": 1, 2024-09-15T20:33:38.392 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables2": 1, 2024-09-15T20:33:38.392 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v2_rules": 0, 2024-09-15T20:33:38.393 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables3": 1, 2024-09-15T20:33:38.393 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v3_rules": 0, 2024-09-15T20:33:38.393 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v4_buckets": 1, 2024-09-15T20:33:38.393 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables5": 1, 2024-09-15T20:33:38.393 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v5_rules": 0, 2024-09-15T20:33:38.393 INFO:tasks.workunit.client.0.smithi139.stdout: "has_msr_rules": 0 2024-09-15T20:33:38.393 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.393 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_args": {} 2024-09-15T20:33:38.393 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.393 INFO:tasks.workunit.client.0.smithi139.stdout: "fsmap": { 2024-09-15T20:33:38.393 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 1, 2024-09-15T20:33:38.393 INFO:tasks.workunit.client.0.smithi139.stdout: "btime": "2024-09-15T20:29:28:939760+0000", 2024-09-15T20:33:38.393 INFO:tasks.workunit.client.0.smithi139.stdout: "default_fscid": -1, 2024-09-15T20:33:38.394 INFO:tasks.workunit.client.0.smithi139.stdout: "compat": { 2024-09-15T20:33:38.394 INFO:tasks.workunit.client.0.smithi139.stdout: "compat": {}, 2024-09-15T20:33:38.394 INFO:tasks.workunit.client.0.smithi139.stdout: "ro_compat": {}, 2024-09-15T20:33:38.394 INFO:tasks.workunit.client.0.smithi139.stdout: "incompat": { 2024-09-15T20:33:38.394 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_1": "base v0.20", 2024-09-15T20:33:38.394 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_2": "client writeable ranges", 2024-09-15T20:33:38.394 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_3": "default file layouts on dirs", 2024-09-15T20:33:38.394 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_4": "dir inode in separate object", 2024-09-15T20:33:38.394 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_5": "mds uses versioned encoding", 2024-09-15T20:33:38.394 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-15T20:33:38.394 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_8": "no anchor table", 2024-09-15T20:33:38.394 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_9": "file layout v2", 2024-09-15T20:33:38.394 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_10": "snaprealm v2", 2024-09-15T20:33:38.394 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_11": "minor log segments", 2024-09-15T20:33:38.395 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_12": "quiesce subvolumes" 2024-09-15T20:33:38.395 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.395 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.395 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_flags": { 2024-09-15T20:33:38.395 INFO:tasks.workunit.client.0.smithi139.stdout: "enable_multiple": true, 2024-09-15T20:33:38.395 INFO:tasks.workunit.client.0.smithi139.stdout: "ever_enabled_multiple": true 2024-09-15T20:33:38.395 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.395 INFO:tasks.workunit.client.0.smithi139.stdout: "standbys": [], 2024-09-15T20:33:38.395 INFO:tasks.workunit.client.0.smithi139.stdout: "filesystems": [] 2024-09-15T20:33:38.395 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.395 INFO:tasks.workunit.client.0.smithi139.stdout: "mdsmap_first_committed": 1, 2024-09-15T20:33:38.395 INFO:tasks.workunit.client.0.smithi139.stdout: "mdsmap_last_committed": 1, 2024-09-15T20:33:38.395 INFO:tasks.workunit.client.0.smithi139.stdout: "auth": { 2024-09-15T20:33:38.395 INFO:tasks.workunit.client.0.smithi139.stdout: "first_committed": 1, 2024-09-15T20:33:38.396 INFO:tasks.workunit.client.0.smithi139.stdout: "last_committed": 5, 2024-09-15T20:33:38.396 INFO:tasks.workunit.client.0.smithi139.stdout: "num_secrets": 3 2024-09-15T20:33:38.396 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.396 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg": 1, 2024-09-15T20:33:38.396 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_active": 1, 2024-09-15T20:33:38.396 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_unknown": 0, 2024-09-15T20:33:38.396 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osd": 3, 2024-09-15T20:33:38.396 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_sum": { 2024-09-15T20:33:38.396 INFO:tasks.workunit.client.0.smithi139.stdout: "stat_sum": { 2024-09-15T20:33:38.396 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes": 62050, 2024-09-15T20:33:38.396 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects": 100, 2024-09-15T20:33:38.396 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_clones": 0, 2024-09-15T20:33:38.396 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_copies": 300, 2024-09-15T20:33:38.397 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T20:33:38.397 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing": 0, 2024-09-15T20:33:38.397 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_degraded": 100, 2024-09-15T20:33:38.397 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_misplaced": 0, 2024-09-15T20:33:38.397 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_unfound": 0, 2024-09-15T20:33:38.397 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_dirty": 100, 2024-09-15T20:33:38.397 INFO:tasks.workunit.client.0.smithi139.stdout: "num_whiteouts": 0, 2024-09-15T20:33:38.397 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read": 0, 2024-09-15T20:33:38.397 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read_kb": 0, 2024-09-15T20:33:38.397 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write": 101, 2024-09-15T20:33:38.397 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write_kb": 101, 2024-09-15T20:33:38.397 INFO:tasks.workunit.client.0.smithi139.stdout: "num_scrub_errors": 0, 2024-09-15T20:33:38.397 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T20:33:38.398 INFO:tasks.workunit.client.0.smithi139.stdout: "num_deep_scrub_errors": 0, 2024-09-15T20:33:38.398 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_recovered": 0, 2024-09-15T20:33:38.398 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_recovered": 0, 2024-09-15T20:33:38.398 INFO:tasks.workunit.client.0.smithi139.stdout: "num_keys_recovered": 0, 2024-09-15T20:33:38.398 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_omap": 0, 2024-09-15T20:33:38.398 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T20:33:38.398 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T20:33:38.398 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush": 0, 2024-09-15T20:33:38.398 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_kb": 0, 2024-09-15T20:33:38.398 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict": 0, 2024-09-15T20:33:38.398 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_kb": 0, 2024-09-15T20:33:38.398 INFO:tasks.workunit.client.0.smithi139.stdout: "num_promote": 0, 2024-09-15T20:33:38.398 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_high": 0, 2024-09-15T20:33:38.398 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_low": 0, 2024-09-15T20:33:38.399 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_some": 0, 2024-09-15T20:33:38.399 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_full": 0, 2024-09-15T20:33:38.399 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_pinned": 0, 2024-09-15T20:33:38.399 INFO:tasks.workunit.client.0.smithi139.stdout: "num_legacy_snapsets": 0, 2024-09-15T20:33:38.399 INFO:tasks.workunit.client.0.smithi139.stdout: "num_large_omap_objects": 0, 2024-09-15T20:33:38.399 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_manifest": 0, 2024-09-15T20:33:38.399 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_bytes": 0, 2024-09-15T20:33:38.399 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_keys": 0, 2024-09-15T20:33:38.399 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_repaired": 0 2024-09-15T20:33:38.399 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.399 INFO:tasks.workunit.client.0.smithi139.stdout: "store_stats": { 2024-09-15T20:33:38.399 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 0, 2024-09-15T20:33:38.399 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 0, 2024-09-15T20:33:38.400 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:33:38.400 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 0, 2024-09-15T20:33:38.400 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 0, 2024-09-15T20:33:38.400 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:33:38.400 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:33:38.400 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:33:38.400 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 0, 2024-09-15T20:33:38.400 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 0 2024-09-15T20:33:38.400 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.400 INFO:tasks.workunit.client.0.smithi139.stdout: "log_size": 17, 2024-09-15T20:33:38.400 INFO:tasks.workunit.client.0.smithi139.stdout: "ondisk_log_size": 17, 2024-09-15T20:33:38.400 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 2, 2024-09-15T20:33:38.400 INFO:tasks.workunit.client.0.smithi139.stdout: "acting": 2, 2024-09-15T20:33:38.401 INFO:tasks.workunit.client.0.smithi139.stdout: "num_store_stats": 0 2024-09-15T20:33:38.401 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.401 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_sum": { 2024-09-15T20:33:38.401 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 0, 2024-09-15T20:33:38.401 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 0, 2024-09-15T20:33:38.401 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pgs": 3, 2024-09-15T20:33:38.401 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osds": 3, 2024-09-15T20:33:38.401 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_osds": 3, 2024-09-15T20:33:38.401 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T20:33:38.401 INFO:tasks.workunit.client.0.smithi139.stdout: "kb": 314572800, 2024-09-15T20:33:38.401 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used": 83684, 2024-09-15T20:33:38.401 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_data": 2120, 2024-09-15T20:33:38.401 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_omap": 10, 2024-09-15T20:33:38.401 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_meta": 81461, 2024-09-15T20:33:38.401 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_avail": 314489116, 2024-09-15T20:33:38.402 INFO:tasks.workunit.client.0.smithi139.stdout: "statfs": { 2024-09-15T20:33:38.402 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 322122547200, 2024-09-15T20:33:38.402 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 322036854784, 2024-09-15T20:33:38.402 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:33:38.402 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 2170880, 2024-09-15T20:33:38.402 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 465898, 2024-09-15T20:33:38.402 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:33:38.402 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:33:38.402 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:33:38.402 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 10396, 2024-09-15T20:33:38.402 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 83416932 2024-09-15T20:33:38.402 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.402 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_peers": [], 2024-09-15T20:33:38.402 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_trim_queue_len": 0, 2024-09-15T20:33:38.403 INFO:tasks.workunit.client.0.smithi139.stdout: "num_snap_trimming": 0, 2024-09-15T20:33:38.403 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shards_repaired": 0, 2024-09-15T20:33:38.403 INFO:tasks.workunit.client.0.smithi139.stdout: "op_queue_age_hist": { 2024-09-15T20:33:38.403 INFO:tasks.workunit.client.0.smithi139.stdout: "histogram": [], 2024-09-15T20:33:38.403 INFO:tasks.workunit.client.0.smithi139.stdout: "upper_bound": 1 2024-09-15T20:33:38.403 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.403 INFO:tasks.workunit.client.0.smithi139.stdout: "perf_stat": { 2024-09-15T20:33:38.403 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ms": 0, 2024-09-15T20:33:38.403 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ms": 0, 2024-09-15T20:33:38.403 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ns": 0, 2024-09-15T20:33:38.403 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ns": 0 2024-09-15T20:33:38.403 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.403 INFO:tasks.workunit.client.0.smithi139.stdout: "alerts": [], 2024-09-15T20:33:38.403 INFO:tasks.workunit.client.0.smithi139.stdout: "network_ping_times": [] 2024-09-15T20:33:38.404 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.404 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_sum_by_class": { 2024-09-15T20:33:38.404 INFO:tasks.workunit.client.0.smithi139.stdout: "hdd": { 2024-09-15T20:33:38.404 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 0, 2024-09-15T20:33:38.404 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 0, 2024-09-15T20:33:38.404 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pgs": 3, 2024-09-15T20:33:38.404 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osds": 3, 2024-09-15T20:33:38.404 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_osds": 3, 2024-09-15T20:33:38.404 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T20:33:38.404 INFO:tasks.workunit.client.0.smithi139.stdout: "kb": 314572800, 2024-09-15T20:33:38.404 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used": 83684, 2024-09-15T20:33:38.404 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_data": 2120, 2024-09-15T20:33:38.404 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_omap": 10, 2024-09-15T20:33:38.404 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_meta": 81461, 2024-09-15T20:33:38.405 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_avail": 314489116, 2024-09-15T20:33:38.405 INFO:tasks.workunit.client.0.smithi139.stdout: "statfs": { 2024-09-15T20:33:38.405 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 322122547200, 2024-09-15T20:33:38.405 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 322036854784, 2024-09-15T20:33:38.405 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:33:38.405 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 2170880, 2024-09-15T20:33:38.405 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 465898, 2024-09-15T20:33:38.405 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:33:38.405 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:33:38.405 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:33:38.405 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 10396, 2024-09-15T20:33:38.405 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 83416932 2024-09-15T20:33:38.405 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.406 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_peers": [], 2024-09-15T20:33:38.406 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_trim_queue_len": 0, 2024-09-15T20:33:38.406 INFO:tasks.workunit.client.0.smithi139.stdout: "num_snap_trimming": 0, 2024-09-15T20:33:38.406 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shards_repaired": 0, 2024-09-15T20:33:38.406 INFO:tasks.workunit.client.0.smithi139.stdout: "op_queue_age_hist": { 2024-09-15T20:33:38.406 INFO:tasks.workunit.client.0.smithi139.stdout: "histogram": [], 2024-09-15T20:33:38.406 INFO:tasks.workunit.client.0.smithi139.stdout: "upper_bound": 1 2024-09-15T20:33:38.406 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.406 INFO:tasks.workunit.client.0.smithi139.stdout: "perf_stat": { 2024-09-15T20:33:38.406 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ms": 0, 2024-09-15T20:33:38.406 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ms": 0, 2024-09-15T20:33:38.406 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ns": 0, 2024-09-15T20:33:38.406 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ns": 0 2024-09-15T20:33:38.407 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.407 INFO:tasks.workunit.client.0.smithi139.stdout: "alerts": [], 2024-09-15T20:33:38.407 INFO:tasks.workunit.client.0.smithi139.stdout: "network_ping_times": [] 2024-09-15T20:33:38.407 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.407 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.407 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_stats": [ 2024-09-15T20:33:38.407 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.407 INFO:tasks.workunit.client.0.smithi139.stdout: "poolid": 1, 2024-09-15T20:33:38.407 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg": 1, 2024-09-15T20:33:38.407 INFO:tasks.workunit.client.0.smithi139.stdout: "stat_sum": { 2024-09-15T20:33:38.407 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes": 62050, 2024-09-15T20:33:38.407 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects": 100, 2024-09-15T20:33:38.407 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_clones": 0, 2024-09-15T20:33:38.407 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_copies": 300, 2024-09-15T20:33:38.408 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T20:33:38.408 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing": 0, 2024-09-15T20:33:38.408 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_degraded": 100, 2024-09-15T20:33:38.408 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_misplaced": 0, 2024-09-15T20:33:38.408 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_unfound": 0, 2024-09-15T20:33:38.408 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_dirty": 100, 2024-09-15T20:33:38.408 INFO:tasks.workunit.client.0.smithi139.stdout: "num_whiteouts": 0, 2024-09-15T20:33:38.408 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read": 0, 2024-09-15T20:33:38.408 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read_kb": 0, 2024-09-15T20:33:38.408 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write": 101, 2024-09-15T20:33:38.408 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write_kb": 101, 2024-09-15T20:33:38.408 INFO:tasks.workunit.client.0.smithi139.stdout: "num_scrub_errors": 0, 2024-09-15T20:33:38.408 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T20:33:38.408 INFO:tasks.workunit.client.0.smithi139.stdout: "num_deep_scrub_errors": 0, 2024-09-15T20:33:38.409 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_recovered": 0, 2024-09-15T20:33:38.409 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_recovered": 0, 2024-09-15T20:33:38.409 INFO:tasks.workunit.client.0.smithi139.stdout: "num_keys_recovered": 0, 2024-09-15T20:33:38.409 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_omap": 0, 2024-09-15T20:33:38.409 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T20:33:38.409 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T20:33:38.409 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush": 0, 2024-09-15T20:33:38.409 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_kb": 0, 2024-09-15T20:33:38.409 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict": 0, 2024-09-15T20:33:38.409 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_kb": 0, 2024-09-15T20:33:38.409 INFO:tasks.workunit.client.0.smithi139.stdout: "num_promote": 0, 2024-09-15T20:33:38.409 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_high": 0, 2024-09-15T20:33:38.409 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_low": 0, 2024-09-15T20:33:38.409 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_some": 0, 2024-09-15T20:33:38.410 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_full": 0, 2024-09-15T20:33:38.410 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_pinned": 0, 2024-09-15T20:33:38.410 INFO:tasks.workunit.client.0.smithi139.stdout: "num_legacy_snapsets": 0, 2024-09-15T20:33:38.410 INFO:tasks.workunit.client.0.smithi139.stdout: "num_large_omap_objects": 0, 2024-09-15T20:33:38.410 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_manifest": 0, 2024-09-15T20:33:38.410 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_bytes": 0, 2024-09-15T20:33:38.410 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_keys": 0, 2024-09-15T20:33:38.410 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_repaired": 0 2024-09-15T20:33:38.410 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.410 INFO:tasks.workunit.client.0.smithi139.stdout: "store_stats": { 2024-09-15T20:33:38.410 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 0, 2024-09-15T20:33:38.410 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 0, 2024-09-15T20:33:38.410 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:33:38.411 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 1220608, 2024-09-15T20:33:38.411 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 185266, 2024-09-15T20:33:38.411 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:33:38.411 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:33:38.411 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:33:38.411 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 0, 2024-09-15T20:33:38.411 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 0 2024-09-15T20:33:38.411 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.411 INFO:tasks.workunit.client.0.smithi139.stdout: "log_size": 17, 2024-09-15T20:33:38.411 INFO:tasks.workunit.client.0.smithi139.stdout: "ondisk_log_size": 17, 2024-09-15T20:33:38.411 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 2, 2024-09-15T20:33:38.411 INFO:tasks.workunit.client.0.smithi139.stdout: "acting": 2, 2024-09-15T20:33:38.411 INFO:tasks.workunit.client.0.smithi139.stdout: "num_store_stats": 3 2024-09-15T20:33:38.411 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.411 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:33:38.412 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_stats": [ 2024-09-15T20:33:38.412 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.412 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:33:38.412 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 154618822677 2024-09-15T20:33:38.412 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.412 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.412 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:33:38.412 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 42949672986 2024-09-15T20:33:38.412 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.412 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.412 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:33:38.412 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 163208757268 2024-09-15T20:33:38.412 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.413 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:33:38.413 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_by_state": [ 2024-09-15T20:33:38.413 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.413 INFO:tasks.workunit.client.0.smithi139.stdout: "state": "active+undersized+degraded", 2024-09-15T20:33:38.413 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 1 2024-09-15T20:33:38.413 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.413 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:33:38.413 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_by_osd": [ 2024-09-15T20:33:38.413 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.413 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:33:38.413 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 0, 2024-09-15T20:33:38.413 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 1, 2024-09-15T20:33:38.413 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:33:38.414 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.414 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.414 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:33:38.414 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 1, 2024-09-15T20:33:38.414 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 1, 2024-09-15T20:33:38.414 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:33:38.414 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.414 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.414 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:33:38.414 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 0, 2024-09-15T20:33:38.414 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 0, 2024-09-15T20:33:38.414 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:33:38.414 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.415 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:33:38.415 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snaps": [ 2024-09-15T20:33:38.415 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:33:38.415 INFO:tasks.workunit.client.0.smithi139.stdout: "pool": 1, 2024-09-15T20:33:38.415 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snaps": {} 2024-09-15T20:33:38.415 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.415 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:33:38.415 INFO:tasks.workunit.client.0.smithi139.stdout: "servicemap": { 2024-09-15T20:33:38.415 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 7, 2024-09-15T20:33:38.415 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:33:20.540416+0000", 2024-09-15T20:33:38.415 INFO:tasks.workunit.client.0.smithi139.stdout: "services": {} 2024-09-15T20:33:38.415 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:33:38.416 INFO:tasks.workunit.client.0.smithi139.stdout: "mgrstat_first_committed": 1, 2024-09-15T20:33:38.416 INFO:tasks.workunit.client.0.smithi139.stdout: "mgrstat_last_committed": 113, 2024-09-15T20:33:38.416 INFO:tasks.workunit.client.0.smithi139.stdout: "logm_first_committed": 1, 2024-09-15T20:33:38.416 INFO:tasks.workunit.client.0.smithi139.stdout: "logm_last_committed": 468, 2024-09-15T20:33:38.416 INFO:tasks.workunit.client.0.smithi139.stdout: "paxos": { 2024-09-15T20:33:38.416 INFO:tasks.workunit.client.0.smithi139.stdout: "first_committed": 1, 2024-09-15T20:33:38.416 INFO:tasks.workunit.client.0.smithi139.stdout: "last_committed": 630, 2024-09-15T20:33:38.416 INFO:tasks.workunit.client.0.smithi139.stdout: "last_pn": 0, 2024-09-15T20:33:38.416 INFO:tasks.workunit.client.0.smithi139.stdout: "accepted_pn": 0 2024-09-15T20:33:38.416 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:33:38.416 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:33:38.417 INFO:tasks.workunit.client.0.smithi139.stderr:report 1019545919 2024-09-15T20:33:38.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-15T20:33:38.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2024-09-15T20:33:38.417 INFO:tasks.workunit.client.0.smithi139.stdout:delay recovery 2024-09-15T20:33:38.418 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2024-09-15T20:33:38.761 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:33:38.761 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-15T20:32:21.773889+0000 42'101 42:131 [0,2] 0 [0,2] 0 0'0 2024-09-15T20:31:19.998451+0000 0'0 2024-09-15T20:31:19.998451+0000 0 0 periodic scrub scheduled @ 2024-09-16T23:16:44.266472+0000 0 0 2024-09-15T20:33:38.761 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:33:38.761 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:33:38.762 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:33:38.777 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-09-15T20:33:38.778 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2024-09-15T20:33:38.778 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T20:33:38.778 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T20:33:38.779 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:33:38.779 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:33:38.779 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:33:38.779 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.0.asok 2024-09-15T20:33:38.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-09-15T20:33:38.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.0.asok set_recovery_delay 100000 2024-09-15T20:33:38.891 INFO:tasks.workunit.client.0.smithi139.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T20:33:38.903 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-09-15T20:33:38.904 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2024-09-15T20:33:38.904 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-15T20:33:38.904 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-15T20:33:38.908 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:33:38.908 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:33:38.908 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:33:38.909 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.2.asok 2024-09-15T20:33:38.909 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-09-15T20:33:38.909 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.2.asok set_recovery_delay 100000 2024-09-15T20:33:39.015 INFO:tasks.workunit.client.0.smithi139.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T20:33:39.027 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2024-09-15T20:33:39.027 INFO:tasks.workunit.client.0.smithi139.stdout:revive divergent 1 2024-09-15T20:33:39.027 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2024-09-15T20:33:39.425 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:33:39.425 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-15T20:32:21.773889+0000 42'101 42:131 [0,2] 0 [0,2] 0 0'0 2024-09-15T20:31:19.998451+0000 0'0 2024-09-15T20:31:19.998451+0000 0 0 periodic scrub scheduled @ 2024-09-16T23:16:44.266472+0000 0 0 2024-09-15T20:33:39.425 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:33:39.425 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:33:39.425 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:33:39.440 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2024-09-15T20:33:40.105 INFO:tasks.workunit.client.0.smithi139.stderr:noup is set 2024-09-15T20:33:40.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-09-15T20:33:40.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-15T20:33:40.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:33:40.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:33:40.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:33:40.126 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-15T20:33:40.126 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:33:40.126 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:33:40.126 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:33:40.126 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:33:40.126 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T20:33:40.126 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T20:33:40.126 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:33:40.126 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:33:40.126 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:33:40.126 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:33:40.126 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:33:40.126 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:33:40.127 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:33:40.128 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:33:40.128 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:33:40.128 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:33:40.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:33:40.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:33:40.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:33:40.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:33:40.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:33:40.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:33:40.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:33:40.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:33:40.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:33:40.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:33:40.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-15T20:33:40.132 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.1 2024-09-15T20:33:40.132 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:33:40.133 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:33:40.133 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-15T20:33:40.133 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:33:40.133 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:33:40.133 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:33:40.133 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:33:40.169 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:33:40.167+0000 7fadee6197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:33:40.171 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:33:40.167+0000 7fadee6197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:33:40.172 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:33:40.171+0000 7fadee6197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:33:40.589 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-15T20:33:40.589 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2024-09-15T20:33:41.394 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:33:41.391+0000 7fadee6197c0 -1 Falling back to public interface 2024-09-15T20:33:44.290 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:33:44.287+0000 7fadee6197c0 -1 osd.1 29 log_to_monitors true 2024-09-15T20:33:45.590 INFO:tasks.workunit.client.0.smithi139.stdout:delay recovery divergent 2024-09-15T20:33:45.590 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2024-09-15T20:33:45.591 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2024-09-15T20:33:46.023 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:33:46.024 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-15T20:32:21.773889+0000 42'101 44:135 [0,2] 0 [0,2] 0 0'0 2024-09-15T20:31:19.998451+0000 0'0 2024-09-15T20:31:19.998451+0000 0 0 periodic scrub scheduled @ 2024-09-16T23:16:44.266472+0000 0 0 2024-09-15T20:33:46.024 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:33:46.024 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:33:46.024 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:33:46.042 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2024-09-15T20:33:46.042 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T20:33:46.042 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T20:33:46.043 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:33:46.043 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:33:46.043 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:33:46.043 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.1.asok 2024-09-15T20:33:46.043 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2024-09-15T20:33:46.043 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.1.asok set_recovery_delay 100000 2024-09-15T20:33:46.164 INFO:tasks.workunit.client.0.smithi139.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T20:33:46.177 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2024-09-15T20:33:46.867 INFO:tasks.workunit.client.0.smithi139.stderr:noup is unset 2024-09-15T20:33:46.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2024-09-15T20:33:46.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:33:46.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:33:46.889 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:33:46.890 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:33:46.890 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:33:46.890 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:46.890 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:33:46.890 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:46.890 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:33:47.536 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 36 up_thru 41 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/1803718560,v1:127.0.0.1:6803/1803718560] [v2:127.0.0.1:6804/1803718560,v1:127.0.0.1:6805/1803718560] exists,up 457b5e59-ab0b-4ff7-8ba7-80b5ae6b10f2 2024-09-15T20:33:47.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:33:47.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:33:47.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:33:47.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2024-09-15T20:33:47.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:33:47.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:33:47.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:33:47.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:33:47.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:47.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:33:47.539 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:33:47.539 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:47.539 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:33:48.009 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/1101771588,v1:127.0.0.1:6819/1101771588] [v2:127.0.0.1:6820/1101771588,v1:127.0.0.1:6821/1101771588] exists,up 037d24c1-96ec-4453-b5af-a66b6de4e720 2024-09-15T20:33:48.011 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:33:48.011 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:33:48.011 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:33:48.011 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2024-09-15T20:33:48.011 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:33:48.011 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:33:48.011 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:33:48.012 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:33:48.012 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:48.012 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:33:48.012 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:33:48.012 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:48.012 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:33:48.512 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/3470466229,v1:127.0.0.1:6811/3470466229] [v2:127.0.0.1:6812/3470466229,v1:127.0.0.1:6813/3470466229] exists,up 24e3e077-bb9a-4ae3-9be6-20ce6b775ae4 2024-09-15T20:33:48.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:33:48.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:33:48.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:33:48.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2024-09-15T20:33:48.936 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:33:48.936 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-15T20:32:21.773889+0000 42'101 44:135 [0,2] 0 [0,2] 0 0'0 2024-09-15T20:31:19.998451+0000 0'0 2024-09-15T20:31:19.998451+0000 0 0 periodic scrub scheduled @ 2024-09-16T23:16:44.266472+0000 0 0 2024-09-15T20:33:48.936 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:33:48.936 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:33:48.936 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:33:48.953 INFO:tasks.workunit.client.0.smithi139.stdout:wait for peering 2024-09-15T20:33:48.954 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2024-09-15T20:33:48.954 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2024-09-15T20:33:49.380 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:33:49.380 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-15T20:32:21.773889+0000 42'101 44:135 [0,2] 0 [0,2] 0 0'0 2024-09-15T20:31:19.998451+0000 0'0 2024-09-15T20:31:19.998451+0000 0 0 periodic scrub scheduled @ 2024-09-16T23:16:44.266472+0000 0 0 2024-09-15T20:33:49.380 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:33:49.380 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:33:49.380 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:33:49.396 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2024-09-15T20:33:49.503 INFO:tasks.workunit.client.0.smithi139.stdout:killing divergent 1 2024-09-15T20:33:49.504 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2024-09-15T20:33:49.504 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2024-09-15T20:33:49.927 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:33:49.927 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-15T20:32:21.773889+0000 42'101 44:135 [0,2] 0 [0,2] 0 0'0 2024-09-15T20:31:19.998451+0000 0'0 2024-09-15T20:31:19.998451+0000 0 0 periodic scrub scheduled @ 2024-09-16T23:16:44.266472+0000 0 0 2024-09-15T20:33:49.928 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:33:49.928 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:33:49.928 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:33:49.943 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2024-09-15T20:33:49.944 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:33:49.944 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:33:49.944 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:33:49.944 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:33:49.944 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:33:50.051 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:33:50.051 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.27419.out 2024-09-15T20:33:50.051 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.27419.out 2024-09-15T20:33:50.052 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-15T20:33:50.052 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:33:50.052 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:33:50.052 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:33:50.052 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-15T20:33:50.052 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.27419.out 2024-09-15T20:33:51.465 INFO:tasks.workunit.client.0.smithi139.stderr:Exporting 1.0 info 1.0( v 48'102 lc 0'0 (29'90,48'102] local-lis/les=47/48 n=101 ec=23/23 lis/c=47/23 les/c/f=48/24/0 sis=47) 2024-09-15T20:33:51.467 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:023dd3d2:::existing_12:head# 2024-09-15T20:33:51.467 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:04c40a0c:::existing_93:head# 2024-09-15T20:33:51.468 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:064ff5fb:::existing_36:head# 2024-09-15T20:33:51.468 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:091f9f1d:::existing_94:head# 2024-09-15T20:33:51.469 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:0a865a93:::existing_54:head# 2024-09-15T20:33:51.469 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:0fed513c:::existing_41:head# 2024-09-15T20:33:51.470 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:13b9e0b8:::existing_92:head# 2024-09-15T20:33:51.470 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:169157fa:::existing_81:head# 2024-09-15T20:33:51.471 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:17731382:::existing_58:head# 2024-09-15T20:33:51.471 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:18a45a39:::existing_43:head# 2024-09-15T20:33:51.472 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:1c160f01:::existing_66:head# 2024-09-15T20:33:51.472 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:1dd6c249:::existing_97:head# 2024-09-15T20:33:51.473 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:227e0c77:::existing_7:head# 2024-09-15T20:33:51.473 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:23e1b063:::existing_59:head# 2024-09-15T20:33:51.474 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:28566edb:::existing_78:head# 2024-09-15T20:33:51.474 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:2a101182:::existing_20:head# 2024-09-15T20:33:51.475 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:2d1519da:::existing_96:head# 2024-09-15T20:33:51.475 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:3078b87a:::existing_86:head# 2024-09-15T20:33:51.476 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:30bde01b:::existing_15:head# 2024-09-15T20:33:51.476 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:317a7e19:::existing_56:head# 2024-09-15T20:33:51.477 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:38b9c7fa:::existing_33:head# 2024-09-15T20:33:51.477 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:3e16d84b:::existing_48:head# 2024-09-15T20:33:51.478 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:45fe8203:::existing_75:head# 2024-09-15T20:33:51.478 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:4e82c9ba:::existing_50:head# 2024-09-15T20:33:51.479 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:5298dcdb:::existing_57:head# 2024-09-15T20:33:51.479 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:52ebaa44:::existing_32:head# 2024-09-15T20:33:51.480 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:53588912:::existing_9:head# 2024-09-15T20:33:51.480 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:5b7e2de6:::existing_84:head# 2024-09-15T20:33:51.481 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:5da0255c:::existing_13:head# 2024-09-15T20:33:51.481 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:5da2dab8:::existing_53:head# 2024-09-15T20:33:51.482 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:5ee3514b:::existing_30:head# 2024-09-15T20:33:51.482 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:602f83fe:::foo:head# 2024-09-15T20:33:51.482 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:60426cc7:::existing_62:head# 2024-09-15T20:33:51.483 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:628918ca:::existing_79:head# 2024-09-15T20:33:51.483 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:63f8ab2d:::existing_77:head# 2024-09-15T20:33:51.484 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:67e1780f:::existing_27:head# 2024-09-15T20:33:51.484 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:68143ca1:::existing_71:head# 2024-09-15T20:33:51.485 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:6bac9509:::existing_29:head# 2024-09-15T20:33:51.485 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:6bbec798:::existing_45:head# 2024-09-15T20:33:51.486 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:6cb23a40:::existing_65:head# 2024-09-15T20:33:51.486 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:7055ec5e:::existing_83:head# 2024-09-15T20:33:51.486 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:70a54e2b:::existing_87:head# 2024-09-15T20:33:51.487 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:750ace21:::existing_19:head# 2024-09-15T20:33:51.487 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:78623b1d:::existing_6:head# 2024-09-15T20:33:51.488 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:79881152:::existing_63:head# 2024-09-15T20:33:51.488 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:7a80c0d5:::existing_99:head# 2024-09-15T20:33:51.489 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:81490677:::existing_76:head# 2024-09-15T20:33:51.489 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:85ec8e5d:::existing_85:head# 2024-09-15T20:33:51.489 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:8e2abbf0:::existing_49:head# 2024-09-15T20:33:51.490 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:8ee0fe0f:::existing_52:head# 2024-09-15T20:33:51.490 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:923c5d4d:::existing_5:head# 2024-09-15T20:33:51.491 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:92731e71:::existing_89:head# 2024-09-15T20:33:51.491 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:928205cf:::existing_82:head# 2024-09-15T20:33:51.492 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:93be2267:::existing_24:head# 2024-09-15T20:33:51.492 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:99ceda2c:::existing_44:head# 2024-09-15T20:33:51.493 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:9dd917ee:::existing_26:head# 2024-09-15T20:33:51.493 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:a28805a6:::existing_72:head# 2024-09-15T20:33:51.494 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:a294a8f6:::existing_16:head# 2024-09-15T20:33:51.494 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:a2f5473a:::existing_42:head# 2024-09-15T20:33:51.494 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:a504a533:::existing_90:head# 2024-09-15T20:33:51.495 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:a691337e:::existing_61:head# 2024-09-15T20:33:51.496 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:a6e8d7ac:::existing_28:head# 2024-09-15T20:33:51.496 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:a9fde86e:::existing_18:head# 2024-09-15T20:33:51.496 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:aa309faa:::existing_68:head# 2024-09-15T20:33:51.497 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:ae40b0ed:::existing_31:head# 2024-09-15T20:33:51.497 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:b279b1fe:::existing_10:head# 2024-09-15T20:33:51.498 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:b343e277:::existing_88:head# 2024-09-15T20:33:51.498 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:b7130e90:::existing_80:head# 2024-09-15T20:33:51.499 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:bc6f7309:::existing_70:head# 2024-09-15T20:33:51.499 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:be8489b7:::existing_38:head# 2024-09-15T20:33:51.500 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:beb4e5e3:::existing_47:head# 2024-09-15T20:33:51.500 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:bebec877:::existing_74:head# 2024-09-15T20:33:51.501 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:c04347d8:::existing_60:head# 2024-09-15T20:33:51.501 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:c156033e:::existing_100:head# 2024-09-15T20:33:51.502 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:c19f5a8e:::existing_8:head# 2024-09-15T20:33:51.502 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:c3e6ebe4:::existing_17:head# 2024-09-15T20:33:51.503 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:c6c5a7c2:::existing_39:head# 2024-09-15T20:33:51.503 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:cc1d0abf:::existing_95:head# 2024-09-15T20:33:51.503 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:ccb3865b:::existing_73:head# 2024-09-15T20:33:51.504 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:cd84e19e:::existing_40:head# 2024-09-15T20:33:51.504 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:d2cd64ce:::existing_37:head# 2024-09-15T20:33:51.505 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:d41a25c9:::existing_22:head# 2024-09-15T20:33:51.505 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:d7f088da:::existing_91:head# 2024-09-15T20:33:51.506 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-09-15T20:33:51.506 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-09-15T20:33:51.507 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:dd677bd3:::existing_35:head# 2024-09-15T20:33:51.507 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-09-15T20:33:51.508 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:e0e22077:::existing_21:head# 2024-09-15T20:33:51.508 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:e255b710:::existing_55:head# 2024-09-15T20:33:51.508 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:eff1d5db:::existing_14:head# 2024-09-15T20:33:51.509 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:f1b156bf:::existing_23:head# 2024-09-15T20:33:51.509 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:f5ce4199:::existing_34:head# 2024-09-15T20:33:51.510 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:f6809456:::existing_11:head# 2024-09-15T20:33:51.510 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:f699a2a9:::existing_67:head# 2024-09-15T20:33:51.511 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:f9baac43:::existing_25:head# 2024-09-15T20:33:51.511 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:fb76688e:::existing_51:head# 2024-09-15T20:33:51.512 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:fbb92eec:::existing_69:head# 2024-09-15T20:33:51.512 INFO:tasks.workunit.client.0.smithi139.stderr:Export successful 2024-09-15T20:33:51.512 INFO:tasks.workunit.client.0.smithi139.stdout: marking collection for removal 2024-09-15T20:33:51.512 INFO:tasks.workunit.client.0.smithi139.stdout:setting '_remove' omap key 2024-09-15T20:33:51.513 INFO:tasks.workunit.client.0.smithi139.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-15T20:33:51.673 INFO:tasks.workunit.client.0.smithi139.stderr:Remove successful 2024-09-15T20:33:52.050 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.27419.out 2024-09-15T20:33:52.050 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-15T20:33:52.050 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:33:52.050 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:33:52.050 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:33:52.050 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-15T20:33:52.050 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.27419.out 2024-09-15T20:33:53.320 INFO:tasks.workunit.client.0.smithi139.stdout:get_pg_num_history pg_num_history pg_num_history(e48 pg_nums {1={23=1}} deleted_pools ) 2024-09-15T20:33:53.320 INFO:tasks.workunit.client.0.smithi139.stdout:Importing pgid 1.0 2024-09-15T20:33:53.321 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:023dd3d2:::existing_12:head# 2024-09-15T20:33:53.321 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.384 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:04c40a0c:::existing_93:head# 2024-09-15T20:33:53.385 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.418 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:064ff5fb:::existing_36:head# 2024-09-15T20:33:53.418 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.451 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:091f9f1d:::existing_94:head# 2024-09-15T20:33:53.451 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.485 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:0a865a93:::existing_54:head# 2024-09-15T20:33:53.485 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.518 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:0fed513c:::existing_41:head# 2024-09-15T20:33:53.518 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.552 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:13b9e0b8:::existing_92:head# 2024-09-15T20:33:53.552 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.585 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:169157fa:::existing_81:head# 2024-09-15T20:33:53.585 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.618 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:17731382:::existing_58:head# 2024-09-15T20:33:53.619 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.652 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:18a45a39:::existing_43:head# 2024-09-15T20:33:53.652 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.685 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:1c160f01:::existing_66:head# 2024-09-15T20:33:53.686 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.719 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:1dd6c249:::existing_97:head# 2024-09-15T20:33:53.719 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.744 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:227e0c77:::existing_7:head# 2024-09-15T20:33:53.744 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.769 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:23e1b063:::existing_59:head# 2024-09-15T20:33:53.769 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.794 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:28566edb:::existing_78:head# 2024-09-15T20:33:53.794 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.819 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:2a101182:::existing_20:head# 2024-09-15T20:33:53.819 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.844 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:2d1519da:::existing_96:head# 2024-09-15T20:33:53.845 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.870 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:3078b87a:::existing_86:head# 2024-09-15T20:33:53.870 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.895 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:30bde01b:::existing_15:head# 2024-09-15T20:33:53.895 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.920 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:317a7e19:::existing_56:head# 2024-09-15T20:33:53.920 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.945 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:38b9c7fa:::existing_33:head# 2024-09-15T20:33:53.945 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.970 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:3e16d84b:::existing_48:head# 2024-09-15T20:33:53.970 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:53.995 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:45fe8203:::existing_75:head# 2024-09-15T20:33:53.995 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.020 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:4e82c9ba:::existing_50:head# 2024-09-15T20:33:54.020 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.045 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:5298dcdb:::existing_57:head# 2024-09-15T20:33:54.046 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.071 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:52ebaa44:::existing_32:head# 2024-09-15T20:33:54.071 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.096 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:53588912:::existing_9:head# 2024-09-15T20:33:54.096 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.121 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:5b7e2de6:::existing_84:head# 2024-09-15T20:33:54.121 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.146 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:5da0255c:::existing_13:head# 2024-09-15T20:33:54.146 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.171 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:5da2dab8:::existing_53:head# 2024-09-15T20:33:54.171 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.196 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:5ee3514b:::existing_30:head# 2024-09-15T20:33:54.196 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.221 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:602f83fe:::foo:head# 2024-09-15T20:33:54.221 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.246 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:60426cc7:::existing_62:head# 2024-09-15T20:33:54.247 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.271 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:628918ca:::existing_79:head# 2024-09-15T20:33:54.272 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.296 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:63f8ab2d:::existing_77:head# 2024-09-15T20:33:54.297 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.322 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:67e1780f:::existing_27:head# 2024-09-15T20:33:54.322 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.347 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:68143ca1:::existing_71:head# 2024-09-15T20:33:54.347 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.372 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:6bac9509:::existing_29:head# 2024-09-15T20:33:54.373 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.397 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:6bbec798:::existing_45:head# 2024-09-15T20:33:54.397 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.422 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:6cb23a40:::existing_65:head# 2024-09-15T20:33:54.423 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.448 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:7055ec5e:::existing_83:head# 2024-09-15T20:33:54.448 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.473 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:70a54e2b:::existing_87:head# 2024-09-15T20:33:54.473 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.498 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:750ace21:::existing_19:head# 2024-09-15T20:33:54.498 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.523 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:78623b1d:::existing_6:head# 2024-09-15T20:33:54.523 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.548 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:79881152:::existing_63:head# 2024-09-15T20:33:54.548 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.573 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:7a80c0d5:::existing_99:head# 2024-09-15T20:33:54.573 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.598 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:81490677:::existing_76:head# 2024-09-15T20:33:54.599 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.665 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:85ec8e5d:::existing_85:head# 2024-09-15T20:33:54.665 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.715 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:8e2abbf0:::existing_49:head# 2024-09-15T20:33:54.716 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.741 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:8ee0fe0f:::existing_52:head# 2024-09-15T20:33:54.741 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.766 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:923c5d4d:::existing_5:head# 2024-09-15T20:33:54.766 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.791 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:92731e71:::existing_89:head# 2024-09-15T20:33:54.791 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.816 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:928205cf:::existing_82:head# 2024-09-15T20:33:54.816 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.841 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:93be2267:::existing_24:head# 2024-09-15T20:33:54.841 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.866 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:99ceda2c:::existing_44:head# 2024-09-15T20:33:54.867 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.891 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:9dd917ee:::existing_26:head# 2024-09-15T20:33:54.891 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.917 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:a28805a6:::existing_72:head# 2024-09-15T20:33:54.917 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:54.950 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:a294a8f6:::existing_16:head# 2024-09-15T20:33:54.950 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.009 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:a2f5473a:::existing_42:head# 2024-09-15T20:33:55.009 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.059 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:a504a533:::existing_90:head# 2024-09-15T20:33:55.059 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.118 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:a691337e:::existing_61:head# 2024-09-15T20:33:55.118 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.151 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:a6e8d7ac:::existing_28:head# 2024-09-15T20:33:55.151 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.185 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:a9fde86e:::existing_18:head# 2024-09-15T20:33:55.185 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.218 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:aa309faa:::existing_68:head# 2024-09-15T20:33:55.218 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.251 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:ae40b0ed:::existing_31:head# 2024-09-15T20:33:55.252 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.310 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:b279b1fe:::existing_10:head# 2024-09-15T20:33:55.310 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.343 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:b343e277:::existing_88:head# 2024-09-15T20:33:55.344 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.377 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:b7130e90:::existing_80:head# 2024-09-15T20:33:55.377 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.410 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:bc6f7309:::existing_70:head# 2024-09-15T20:33:55.411 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.444 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:be8489b7:::existing_38:head# 2024-09-15T20:33:55.444 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.477 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:beb4e5e3:::existing_47:head# 2024-09-15T20:33:55.477 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.511 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:bebec877:::existing_74:head# 2024-09-15T20:33:55.511 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.544 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:c04347d8:::existing_60:head# 2024-09-15T20:33:55.544 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.578 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:c156033e:::existing_100:head# 2024-09-15T20:33:55.578 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.611 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:c19f5a8e:::existing_8:head# 2024-09-15T20:33:55.611 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.645 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:c3e6ebe4:::existing_17:head# 2024-09-15T20:33:55.645 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.679 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:c6c5a7c2:::existing_39:head# 2024-09-15T20:33:55.680 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.703 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:cc1d0abf:::existing_95:head# 2024-09-15T20:33:55.703 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.728 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:ccb3865b:::existing_73:head# 2024-09-15T20:33:55.728 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.753 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:cd84e19e:::existing_40:head# 2024-09-15T20:33:55.753 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.779 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:d2cd64ce:::existing_37:head# 2024-09-15T20:33:55.779 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.804 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:d41a25c9:::existing_22:head# 2024-09-15T20:33:55.804 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.829 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:d7f088da:::existing_91:head# 2024-09-15T20:33:55.829 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.854 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-09-15T20:33:55.854 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.879 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-09-15T20:33:55.879 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.904 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:dd677bd3:::existing_35:head# 2024-09-15T20:33:55.904 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.929 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-09-15T20:33:55.929 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.954 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:e0e22077:::existing_21:head# 2024-09-15T20:33:55.954 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:55.979 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:e255b710:::existing_55:head# 2024-09-15T20:33:55.980 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:56.005 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:eff1d5db:::existing_14:head# 2024-09-15T20:33:56.005 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:56.030 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:f1b156bf:::existing_23:head# 2024-09-15T20:33:56.030 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:56.055 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:f5ce4199:::existing_34:head# 2024-09-15T20:33:56.055 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:56.080 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:f6809456:::existing_11:head# 2024-09-15T20:33:56.080 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:56.105 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:f699a2a9:::existing_67:head# 2024-09-15T20:33:56.105 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:56.156 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:f9baac43:::existing_25:head# 2024-09-15T20:33:56.156 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:56.181 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:fb76688e:::existing_51:head# 2024-09-15T20:33:56.181 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:56.206 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:fbb92eec:::existing_69:head# 2024-09-15T20:33:56.206 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:33:56.231 INFO:tasks.workunit.client.0.smithi139.stdout:write_pg epoch 48 info 1.0( v 48'102 lc 0'0 (29'90,48'102] local-lis/les=47/48 n=101 ec=23/23 lis/c=47/23 les/c/f=48/24/0 sis=47) 2024-09-15T20:33:56.256 INFO:tasks.workunit.client.0.smithi139.stdout:Import successful 2024-09-15T20:33:56.797 INFO:tasks.workunit.client.0.smithi139.stdout:reviving divergent 1 2024-09-15T20:33:56.797 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2024-09-15T20:33:56.797 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2024-09-15T20:33:57.211 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:33:57.211 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 101 4 4 0 0 62667 0 0 12 90 12 active+recovering+degraded 2024-09-15T20:33:47.526025+0000 48'102 48:133 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:31:19.998451+0000 0'0 2024-09-15T20:31:19.998451+0000 0 0 no scrub is scheduled 0 0 2024-09-15T20:33:57.211 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:33:57.211 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:33:57.211 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:33:57.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-09-15T20:33:57.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-15T20:33:57.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:33:57.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:33:57.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:33:57.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-15T20:33:57.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:33:57.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:33:57.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:33:57.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:33:57.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T20:33:57.230 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T20:33:57.230 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:33:57.230 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:33:57.230 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:33:57.230 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:33:57.230 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:33:57.230 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:33:57.230 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:33:57.230 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:33:57.230 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:33:57.231 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:33:57.232 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:33:57.232 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:33:57.232 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:33:57.232 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:33:57.232 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:33:57.232 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:33:57.232 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:33:57.232 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:33:57.232 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:33:57.232 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:33:57.232 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-15T20:33:57.233 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.1 2024-09-15T20:33:57.233 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:33:57.234 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:33:57.234 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-15T20:33:57.234 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:33:57.234 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:33:57.234 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:33:57.234 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:33:57.273 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:33:57.271+0000 7fd6afb827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:33:57.275 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:33:57.275+0000 7fd6afb827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:33:57.278 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:33:57.275+0000 7fd6afb827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:33:57.657 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:33:57.657 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:33:57.657 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:33:57.657 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:33:57.658 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:33:57.658 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:33:57.658 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:57.658 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:33:57.658 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:57.658 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:33:58.103 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/1101771588,v1:127.0.0.1:6819/1101771588] [v2:127.0.0.1:6820/1101771588,v1:127.0.0.1:6821/1101771588] exists,up 037d24c1-96ec-4453-b5af-a66b6de4e720 2024-09-15T20:33:58.104 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:33:58.104 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:33:58.104 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:33:58.104 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2024-09-15T20:33:58.104 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:33:58.104 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:33:58.104 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:33:58.104 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:33:58.104 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:33:58.105 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:33:58.105 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:33:58.105 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:33:58.105 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:33:58.338 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:33:58.335+0000 7fd6afb827c0 -1 Falling back to public interface 2024-09-15T20:33:58.555 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/1101771588,v1:127.0.0.1:6819/1101771588] [v2:127.0.0.1:6820/1101771588,v1:127.0.0.1:6821/1101771588] exists,up 037d24c1-96ec-4453-b5af-a66b6de4e720 2024-09-15T20:33:58.556 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:33:58.556 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:33:58.556 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:33:58.556 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2024-09-15T20:34:01.386 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:34:01.383+0000 7fd6afb827c0 -1 osd.1 48 log_to_monitors true 2024-09-15T20:34:18.562 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2024-09-15T20:34:18.562 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T20:34:18.562 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T20:34:18.562 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:34:18.562 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:34:18.562 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:34:18.563 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.1.asok 2024-09-15T20:34:18.563 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2024-09-15T20:34:18.563 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T20:34:18.677 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:34:18.677 INFO:tasks.workunit.client.0.smithi139.stdout: "ops": [], 2024-09-15T20:34:18.677 INFO:tasks.workunit.client.0.smithi139.stdout: "num_ops": 0 2024-09-15T20:34:18.677 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:34:18.689 INFO:tasks.workunit.client.0.smithi139.stdout:allowing recovery 2024-09-15T20:34:18.690 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2024-09-15T20:34:18.690 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2024-09-15T20:34:19.093 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:34:19.093 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-09-15T20:34:03.845732+0000 48'102 53:156 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:31:19.998451+0000 0'0 2024-09-15T20:31:19.998451+0000 0 0 periodic scrub scheduled @ 2024-09-17T00:44:36.397102+0000 0 0 2024-09-15T20:34:19.093 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:34:19.093 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:34:19.094 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:34:19.108 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-15T20:34:19.108 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-15T20:34:19.260 INFO:tasks.workunit.client.0.smithi139.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T20:34:19.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-15T20:34:19.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-15T20:34:19.425 INFO:tasks.workunit.client.0.smithi139.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T20:34:19.439 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-15T20:34:19.439 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-15T20:34:19.579 INFO:tasks.workunit.client.0.smithi139.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T20:34:19.593 INFO:tasks.workunit.client.0.smithi139.stdout:reading divergent objects 2024-09-15T20:34:19.593 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2024-09-15T20:34:19.593 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2024-09-15T20:34:19.990 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:34:19.990 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-09-15T20:34:03.845732+0000 48'102 53:156 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:31:19.998451+0000 0'0 2024-09-15T20:31:19.998451+0000 0 0 periodic scrub scheduled @ 2024-09-17T00:44:36.397102+0000 0 0 2024-09-15T20:34:19.990 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:34:19.990 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:34:19.991 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:34:20.007 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2024-09-15T20:34:20.008 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2024-09-15T20:34:20.009 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T20:34:20.009 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2024-09-15T20:34:20.065 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T20:34:20.065 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2024-09-15T20:34:20.101 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T20:34:20.101 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2024-09-15T20:34:20.137 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T20:34:20.137 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2024-09-15T20:34:20.173 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2024-09-15T20:34:20.174 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-15T20:34:20.174 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2024-09-15T20:34:20.208 INFO:tasks.workunit.client.0.smithi139.stderr:error getting test/newobject_1: (2) No such file or directory 2024-09-15T20:34:20.209 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-15T20:34:20.210 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2024-09-15T20:34:20.243 INFO:tasks.workunit.client.0.smithi139.stderr:error getting test/newobject_2: (2) No such file or directory 2024-09-15T20:34:20.245 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2024-09-15T20:34:20.246 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-15T20:34:20.247 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-15T20:34:20.251 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'104 (29'4) delete 1:70b3bf12:::existing_4:head by client.4630.0:1 2024-09-15T20:31:55.509448+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-15T20:34:20.251 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 20 _merge_object_divergent_entries: keeping 29'104 (29'4) delete 1:70b3bf12:::existing_4:head by client.4630.0:1 2024-09-15T20:31:55.509448+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-15T20:34:20.251 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T20:34:20.251 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'104 last_divergent_update: 29'104 2024-09-15T20:34:20.251 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: more recent entry found: 42'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4752.0:1 2024-09-15T20:33:00.258258+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false, already merged 2024-09-15T20:34:20.252 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4620.0:1 2024-09-15T20:31:55.491446+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-15T20:34:20.252 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 20 _merge_object_divergent_entries: keeping 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4620.0:1 2024-09-15T20:31:55.491446+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-15T20:34:20.252 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T20:34:20.252 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-09-15T20:34:20.252 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-09-15T20:34:20.252 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-09-15T20:34:20.252 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-09-15T20:31:55.514035+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-15T20:34:20.252 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 20 _merge_object_divergent_entries: keeping 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-09-15T20:31:55.514035+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-15T20:34:20.252 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-15T20:34:20.252 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'105 last_divergent_update: 29'105 2024-09-15T20:34:20.252 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-15T20:34:20.253 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-15T20:34:20.253 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2024-09-15T20:34:20.253 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4633.0:1 2024-09-15T20:31:55.514035+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-15T20:34:20.253 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-15T20:34:20.253 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'90 2024-09-15T20:34:20.253 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4625.0:1 2024-09-15T20:31:55.500028+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-15T20:34:20.253 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 20 _merge_object_divergent_entries: keeping 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4625.0:1 2024-09-15T20:31:55.500028+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-15T20:34:20.253 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T20:34:20.254 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'102 last_divergent_update: 29'102 2024-09-15T20:34:20.254 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-09-15T20:34:20.254 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-09-15T20:34:20.254 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2024-09-15T20:34:20.254 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4625.0:1 2024-09-15T20:31:55.500028+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-15T20:34:20.254 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-09-15T20:34:20.254 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'90 2024-09-15T20:34:20.254 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'106 (29'1) modify 1:d858c035:::existing_1:head by client.4635.0:1 2024-09-15T20:31:55.514843+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-15T20:34:20.254 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 20 _merge_object_divergent_entries: keeping 29'106 (29'1) modify 1:d858c035:::existing_1:head by client.4635.0:1 2024-09-15T20:31:55.514843+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-15T20:34:20.255 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T20:34:20.255 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'106 last_divergent_update: 29'106 2024-09-15T20:34:20.255 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-09-15T20:34:20.255 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-09-15T20:34:20.255 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2024-09-15T20:34:20.255 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'106 (29'1) modify 1:d858c035:::existing_1:head by client.4635.0:1 2024-09-15T20:31:55.514843+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-15T20:34:20.255 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-09-15T20:34:20.255 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2024-09-15T20:34:20.255 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4626.0:1 2024-09-15T20:31:55.501799+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-15T20:34:20.255 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 20 _merge_object_divergent_entries: keeping 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4626.0:1 2024-09-15T20:31:55.501799+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-15T20:34:20.255 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T20:34:20.256 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'103 last_divergent_update: 29'103 2024-09-15T20:34:20.256 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-09-15T20:34:20.256 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:33:47.367+0000 7fadd25a3640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-09-15T20:34:20.256 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-15T20:34:20.256 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-15T20:34:20.259 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-15T20:34:20.260 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-15T20:34:20.264 INFO:tasks.workunit.client.0.smithi139.stdout:success 2024-09-15T20:34:20.265 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2024-09-15T20:34:20.265 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.27419.out 2024-09-15T20:34:20.265 INFO:tasks.workunit.client.0.smithi139.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.27419.out': No such file or directory 2024-09-15T20:34:20.265 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2024-09-15T20:34:20.266 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-15T20:34:20.266 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-15T20:34:20.840 INFO:tasks.workunit.client.0.smithi139.stderr:pool 'test' does not exist 2024-09-15T20:34:20.857 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2024-09-15T20:34:20.858 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:34:20.858 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:34:20.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:34:20.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:34:20.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:34:21.170 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:34:21.170 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-15T20:34:21.170 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-15T20:34:21.170 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:34:21.171 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-15T20:34:21.171 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:34:21.171 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:34:21.171 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:34:21.172 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:34:21.172 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:34:21.179 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:34:21.179 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:34:21.179 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:34:21.180 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:34:21.181 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:34:21.181 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:34:21.181 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:34:21.182 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:34:21.182 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:34:21.183 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:34:21.183 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:34:21.183 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:34:21.185 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:34:21.185 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-15T20:34:21.211 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:34:21.211 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:34:21.211 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:34:21.211 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27419 2024-09-15T20:34:21.213 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:34:21.213 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:34:21.213 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-15T20:34:21.213 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-15T20:34:21.213 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-15T20:34:21.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-15T20:34:21.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-15T20:34:21.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:34:21.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-15T20:34:21.214 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:34:21.214 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:34:21.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:34:21.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:34:21.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:34:21.215 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:34:21.215 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:34:21.216 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:34:21.216 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:34:21.217 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:34:21.218 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:34:21.218 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:34:21.218 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:34:21.218 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:34:21.219 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:34:21.219 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:34:21.219 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:34:21.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:34:21.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-15T20:34:21.222 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:34:21.222 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:34:21.222 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:34:21.222 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27419 2024-09-15T20:34:21.223 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:34:21.223 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:34:21.223 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-15T20:34:21.224 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:34:21.224 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:34:21.224 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:34:21.225 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27419 2024-09-15T20:34:21.226 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:34:21.226 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T20:34:21.226 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:34:21.227 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-15T20:34:21.227 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2024-09-15T20:34:21.227 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2024-09-15T20:34:21.227 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2024-09-15T20:34:21.227 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2024-09-15T20:34:21.227 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2024-09-15T20:34:21.227 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2024-09-15T20:34:21.228 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2024-09-15T20:34:21.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2024-09-15T20:34:21.229 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:34:21.229 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:34:21.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2024-09-15T20:34:21.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-15T20:34:21.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T20:34:21.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T20:34:21.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T20:34:21.230 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-15T20:34:21.230 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-15T20:34:21.701 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T20:34:21.701 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:34:21.702 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:34:21.702 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:34:21.702 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:34:21.702 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:34:21.702 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:34:21.703 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T20:34:21.921 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T20:34:21.922 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T20:34:21.923 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:34:21.923 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:34:21.923 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T20:34:21.924 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T20:34:21.924 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:34:21.924 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:34:21.924 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:34:21.924 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:34:21.924 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:34:21.924 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:34:21.925 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-mon.a.asok 2024-09-15T20:34:21.925 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:34:21.925 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27419/ceph-mon.a.asok config get fsid 2024-09-15T20:34:22.040 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T20:34:22.040 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:34:22.040 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:34:22.040 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T20:34:22.041 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T20:34:22.041 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:34:22.041 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:34:22.042 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:34:22.042 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:34:22.042 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:34:22.042 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:34:22.042 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-mon.a.asok 2024-09-15T20:34:22.042 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:34:22.042 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27419/ceph-mon.a.asok config get mon_host 2024-09-15T20:34:22.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2024-09-15T20:34:22.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-15T20:34:22.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T20:34:22.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T20:34:22.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T20:34:22.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-15T20:34:22.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T20:34:23.097 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T20:34:23.097 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:34:23.097 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:34:23.098 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:34:23.098 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:34:23.098 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:34:23.098 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:34:23.099 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:34:23.100 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:34:23.161 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-15T20:34:23.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2024-09-15T20:34:23.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-15T20:34:23.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:34:23.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T20:34:23.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:34:23.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-15T20:34:23.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:34:23.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:34:23.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:34:23.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:34:23.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-15T20:34:23.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-15T20:34:23.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:34:23.163 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:34:23.163 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:34:23.163 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:34:23.163 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:34:23.163 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:34:23.163 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:34:23.163 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:34:23.163 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:34:23.163 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:34:23.163 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:34:23.163 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:34:23.163 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:34:23.163 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:34:23.163 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:34:23.164 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:34:23.164 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:34:23.164 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:34:23.164 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:34:23.164 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:34:23.164 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:34:23.164 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:34:23.164 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-15T20:34:23.165 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:34:23.165 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8ceaad64-e87a-460e-8e98-3caf0f7eff12 2024-09-15T20:34:23.165 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8ceaad64-e87a-460e-8e98-3caf0f7eff12' 2024-09-15T20:34:23.165 INFO:tasks.workunit.client.0.smithi139.stdout:add osd0 8ceaad64-e87a-460e-8e98-3caf0f7eff12 2024-09-15T20:34:23.165 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:34:23.187 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDPROdmSntPCxAAGwOLcR7mcKJGGgxG6eyMiA== 2024-09-15T20:34:23.188 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDPROdmSntPCxAAGwOLcR7mcKJGGgxG6eyMiA=="}' 2024-09-15T20:34:23.188 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8ceaad64-e87a-460e-8e98-3caf0f7eff12 -i td/divergent-priors/0/new.json 2024-09-15T20:34:23.484 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:34:23.499 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-15T20:34:23.500 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPROdmSntPCxAAGwOLcR7mcKJGGgxG6eyMiA== --osd-uuid 8ceaad64-e87a-460e-8e98-3caf0f7eff12 2024-09-15T20:34:23.544 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:34:23.543+0000 7f7b47d337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:34:23.551 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:34:23.547+0000 7f7b47d337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:34:23.552 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:34:23.551+0000 7f7b47d337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:34:23.552 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:34:23.551+0000 7f7b47d337c0 -1 bdev(0x5600d5e91c00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:34:23.553 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:34:23.551+0000 7f7b47d337c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-15T20:34:29.559 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-15T20:34:29.559 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:34:29.560 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd0 key to auth repository 2024-09-15T20:34:29.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T20:34:29.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:34:30.179 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T20:34:30.179 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T20:34:30.180 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:34:30.180 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:34:30.180 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:34:30.180 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:34:30.219 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:34:30.218+0000 7fd49378c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:34:30.221 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:34:30.218+0000 7fd49378c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:34:30.222 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:34:30.218+0000 7fd49378c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:34:30.650 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T20:34:30.650 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:34:30.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:34:30.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:34:30.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:34:30.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:30.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:34:30.651 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:34:30.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:30.652 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:31.127 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:31.306 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:34:31.302+0000 7fd49378c7c0 -1 Falling back to public interface 2024-09-15T20:34:32.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:32.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:32.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:34:32.130 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:34:32.130 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:32.130 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:32.611 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:33.613 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:33.613 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:33.614 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:34:33.614 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:34:33.614 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:33.614 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:33.925 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:34:33.922+0000 7fd49378c7c0 -1 osd.0 0 log_to_monitors true 2024-09-15T20:34:34.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:35.097 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:35.098 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:35.098 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:34:35.098 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:34:35.099 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:35.099 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:35.679 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:36.681 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:36.681 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:36.681 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:34:36.682 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:34:36.682 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:36.682 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:37.114 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:38.116 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:38.116 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:38.116 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:34:38.116 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:34:38.116 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:38.117 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:38.673 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:39.675 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:39.675 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:39.675 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:34:39.676 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:34:39.676 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:39.676 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:40.153 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:41.155 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:41.155 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:41.155 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:34:41.156 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:34:41.156 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:41.156 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:41.738 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:42.739 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:42.739 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:42.739 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:34:42.740 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:34:42.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:42.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:43.190 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:44.192 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:44.192 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:44.192 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:34:44.192 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:34:44.193 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:44.193 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:44.702 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:45.703 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:45.703 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:45.704 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:34:45.704 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:34:45.704 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:45.705 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:46.553 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:47.554 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:47.555 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:47.555 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:34:47.555 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:34:47.555 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:47.555 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:48.033 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:49.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:49.036 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:49.036 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:34:49.036 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:34:49.036 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:49.036 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:50.057 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:51.059 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:51.059 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:51.060 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:34:51.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:34:51.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:51.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:52.013 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:53.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:53.015 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:34:53.015 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:53.015 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:34:53.015 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:53.015 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:53.804 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:54.805 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:54.805 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:54.806 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:34:54.806 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:34:54.806 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:54.806 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:55.525 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:34:56.526 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:34:56.526 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:34:56.527 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:34:56.527 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:34:56.527 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:34:56.527 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:34:57.008 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/233783317,v1:127.0.0.1:6803/233783317] [v2:127.0.0.1:6804/233783317,v1:127.0.0.1:6805/233783317] exists,up 8ceaad64-e87a-460e-8e98-3caf0f7eff12 2024-09-15T20:34:57.012 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:34:57.012 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:34:57.012 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:34:57.012 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-15T20:34:57.012 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2024-09-15T20:34:57.012 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-15T20:34:57.012 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:34:57.012 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T20:34:57.012 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:34:57.012 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-15T20:34:57.013 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:34:57.013 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:34:57.013 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:34:57.013 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:34:57.013 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T20:34:57.013 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T20:34:57.013 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:34:57.013 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:34:57.013 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:34:57.013 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:34:57.013 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:34:57.013 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:34:57.013 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:34:57.013 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:34:57.014 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:34:57.014 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:34:57.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:34:57.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:34:57.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:34:57.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:34:57.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:34:57.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:34:57.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:34:57.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:34:57.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:34:57.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:34:57.015 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:34:57.015 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:34:57.015 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-15T20:34:57.015 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:34:57.015 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d5b37d03-d9a9-4f3f-8c3e-d61cdbdce38e 2024-09-15T20:34:57.015 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 d5b37d03-d9a9-4f3f-8c3e-d61cdbdce38e' 2024-09-15T20:34:57.015 INFO:tasks.workunit.client.0.smithi139.stdout:add osd1 d5b37d03-d9a9-4f3f-8c3e-d61cdbdce38e 2024-09-15T20:34:57.016 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:34:57.041 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDxROdmuk+aAhAAWj8+dbuf/xGDoVLAdtZdFw== 2024-09-15T20:34:57.041 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDxROdmuk+aAhAAWj8+dbuf/xGDoVLAdtZdFw=="}' 2024-09-15T20:34:57.041 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d5b37d03-d9a9-4f3f-8c3e-d61cdbdce38e -i td/divergent-priors/1/new.json 2024-09-15T20:34:57.546 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:34:57.562 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-15T20:34:57.563 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDxROdmuk+aAhAAWj8+dbuf/xGDoVLAdtZdFw== --osd-uuid d5b37d03-d9a9-4f3f-8c3e-d61cdbdce38e 2024-09-15T20:34:57.606 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:34:57.602+0000 7f48eb0df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:34:57.608 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:34:57.606+0000 7f48eb0df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:34:57.610 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:34:57.606+0000 7f48eb0df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:34:57.610 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:34:57.606+0000 7f48eb0df7c0 -1 bdev(0x563e1011fc00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:34:57.610 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:34:57.610+0000 7f48eb0df7c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-15T20:35:03.648 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-15T20:35:03.648 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:35:03.649 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd1 key to auth repository 2024-09-15T20:35:03.649 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T20:35:03.649 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:35:04.258 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.1 2024-09-15T20:35:04.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T20:35:04.259 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:35:04.259 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:35:04.259 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:35:04.259 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:35:04.298 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:35:04.294+0000 7fba44bf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:04.339 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:35:04.338+0000 7fba44bf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:04.341 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:35:04.338+0000 7fba44bf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:04.705 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T20:35:04.705 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:35:04.705 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:35:04.705 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:35:04.705 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:35:04.705 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:04.706 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:35:04.706 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:35:04.706 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:04.706 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:05.190 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:05.433 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:35:05.434+0000 7fba44bf77c0 -1 Falling back to public interface 2024-09-15T20:35:06.192 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:06.192 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:06.192 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:35:06.193 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:35:06.193 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:06.193 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:06.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:07.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:07.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:07.787 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:35:07.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:35:07.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:07.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:08.105 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:35:08.106+0000 7fba44bf77c0 -1 osd.1 0 log_to_monitors true 2024-09-15T20:35:08.292 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:09.294 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:09.294 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:09.294 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:35:09.295 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:35:09.295 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:09.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:09.733 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:10.735 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:10.735 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:10.736 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:35:10.736 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:35:10.736 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:10.736 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:11.294 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:12.295 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:12.295 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:12.296 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:35:12.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:35:12.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:12.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:12.754 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:13.756 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:13.756 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:13.756 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:35:13.757 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:35:13.757 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:13.757 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:14.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:15.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:15.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:15.304 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:35:15.305 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:35:15.305 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:15.305 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:15.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:16.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:16.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:16.787 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:35:16.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:35:16.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:16.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:17.382 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:18.384 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:18.384 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:18.385 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:35:18.385 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:35:18.385 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:18.385 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:18.956 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:19.958 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:19.958 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:19.958 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:35:19.958 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:35:19.959 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:19.959 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:20.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:21.718 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:21.718 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:21.718 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:35:21.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:35:21.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:21.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:22.782 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:23.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:23.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:23.784 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:35:23.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:35:23.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:23.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:24.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:25.281 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:25.281 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:25.282 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:35:25.282 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:35:25.282 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:25.282 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:26.067 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:27.069 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:27.069 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:27.069 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:35:27.070 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:35:27.070 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:27.070 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:27.922 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:28.923 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:28.924 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:28.924 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:35:28.924 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:35:28.924 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:28.925 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:29.686 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:30.688 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:30.688 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:30.689 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:35:30.689 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:35:30.689 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:30.689 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:31.442 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:32.443 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:32.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:32.444 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:35:32.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:35:32.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:32.445 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:35:32.883 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/351107530,v1:127.0.0.1:6811/351107530] [v2:127.0.0.1:6812/351107530,v1:127.0.0.1:6813/351107530] exists,up d5b37d03-d9a9-4f3f-8c3e-d61cdbdce38e 2024-09-15T20:35:32.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:35:32.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:35:32.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:35:32.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-15T20:35:32.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2024-09-15T20:35:32.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-15T20:35:32.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:35:32.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T20:35:32.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:35:32.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-15T20:35:32.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:35:32.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:35:32.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:35:32.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:35:32.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-15T20:35:32.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-15T20:35:32.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:35:32.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:35:32.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:35:32.886 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:35:32.886 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:35:32.886 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:35:32.887 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:35:32.887 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:35:32.887 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:35:32.888 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:35:32.888 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:35:32.888 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:35:32.888 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:35:32.888 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:35:32.888 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:35:32.888 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:35:32.888 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:35:32.888 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:35:32.888 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:35:32.888 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:35:32.888 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:35:32.888 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:35:32.888 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-15T20:35:32.889 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:35:32.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5347fe2a-12cd-44fe-9cd9-0dbc2bc7ebb8 2024-09-15T20:35:32.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 5347fe2a-12cd-44fe-9cd9-0dbc2bc7ebb8' 2024-09-15T20:35:32.890 INFO:tasks.workunit.client.0.smithi139.stdout:add osd2 5347fe2a-12cd-44fe-9cd9-0dbc2bc7ebb8 2024-09-15T20:35:32.890 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:35:32.912 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAURedmAdaLNhAAc2/kRP17Xnt6qsiKehFEZQ== 2024-09-15T20:35:32.912 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAURedmAdaLNhAAc2/kRP17Xnt6qsiKehFEZQ=="}' 2024-09-15T20:35:32.912 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5347fe2a-12cd-44fe-9cd9-0dbc2bc7ebb8 -i td/divergent-priors/2/new.json 2024-09-15T20:35:33.426 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:35:33.442 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-15T20:35:33.443 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAURedmAdaLNhAAc2/kRP17Xnt6qsiKehFEZQ== --osd-uuid 5347fe2a-12cd-44fe-9cd9-0dbc2bc7ebb8 2024-09-15T20:35:33.477 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:35:33.478+0000 7f021d3be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:33.479 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:35:33.482+0000 7f021d3be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:33.480 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:35:33.482+0000 7f021d3be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:33.481 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:35:33.482+0000 7f021d3be7c0 -1 bdev(0x55ae0c2c7c00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-15T20:35:33.481 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:35:33.482+0000 7f021d3be7c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-15T20:35:39.632 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-15T20:35:39.632 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:35:39.634 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd2 key to auth repository 2024-09-15T20:35:39.634 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T20:35:39.634 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:35:40.269 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.2 2024-09-15T20:35:40.270 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T20:35:40.270 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:35:40.270 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:35:40.270 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:35:40.270 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:35:40.309 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:35:40.310+0000 7fe00e0a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:40.313 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:35:40.314+0000 7fe00e0a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:40.314 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:35:40.318+0000 7fe00e0a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:35:40.758 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T20:35:40.758 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:35:40.758 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:35:40.758 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:35:40.758 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:35:40.758 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:40.758 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:35:40.759 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:35:40.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:40.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:41.217 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:41.845 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:35:41.846+0000 7fe00e0a77c0 -1 Falling back to public interface 2024-09-15T20:35:42.219 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:42.219 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:42.219 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:35:42.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:35:42.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:42.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:42.646 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:43.647 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:43.647 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:43.648 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:35:43.648 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:35:43.648 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:43.648 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:44.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:44.697 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:35:44.698+0000 7fe00e0a77c0 -1 osd.2 0 log_to_monitors true 2024-09-15T20:35:45.127 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:45.127 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:45.127 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:35:45.127 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:35:45.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:45.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:45.841 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:46.843 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:46.843 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:46.843 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:35:46.844 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:35:46.844 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:46.844 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:47.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:48.282 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:48.282 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:48.282 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:35:48.283 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:35:48.283 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:48.283 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:48.895 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:49.897 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:49.897 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:49.897 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:35:49.897 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:35:49.898 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:49.898 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:50.474 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:51.476 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:51.476 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:51.476 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:35:51.476 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:35:51.477 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:51.477 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:51.937 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:52.938 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:52.938 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:52.939 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:35:52.939 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:35:52.939 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:52.939 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:53.509 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:54.511 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:54.511 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:54.511 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:35:54.512 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:35:54.512 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:54.512 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:55.075 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:56.076 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:56.076 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:56.077 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:35:56.077 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:35:56.077 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:56.077 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:56.598 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:57.600 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:57.600 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:57.600 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:35:57.601 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:35:57.601 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:57.601 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:35:58.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:35:59.523 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:35:59.523 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:35:59.523 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:35:59.523 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:35:59.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:35:59.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:36:00.370 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:01.371 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:01.371 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:01.372 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:36:01.372 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:36:01.372 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:01.372 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:36:02.181 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:03.183 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:03.183 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:03.183 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:36:03.184 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:36:03.184 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:03.184 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:36:03.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:04.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:04.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:04.977 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:36:04.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:36:04.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:04.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:36:05.701 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:06.703 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:06.703 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:06.703 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:36:06.704 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:36:06.704 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:06.704 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:36:07.381 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:36:08.383 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:36:08.383 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:36:08.383 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:36:08.383 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:36:08.384 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:36:08.384 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:36:08.825 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4199134804,v1:127.0.0.1:6819/4199134804] [v2:127.0.0.1:6820/4199134804,v1:127.0.0.1:6821/4199134804] exists,up 5347fe2a-12cd-44fe-9cd9-0dbc2bc7ebb8 2024-09-15T20:36:08.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:36:08.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:36:08.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:36:08.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2024-09-15T20:36:09.453 INFO:tasks.workunit.client.0.smithi139.stderr:noout is set 2024-09-15T20:36:09.472 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2024-09-15T20:36:10.315 INFO:tasks.workunit.client.0.smithi139.stderr:noin is set 2024-09-15T20:36:10.334 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2024-09-15T20:36:11.093 INFO:tasks.workunit.client.0.smithi139.stderr:nodown is set 2024-09-15T20:36:11.113 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2024-09-15T20:36:11.113 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-15T20:36:11.853 INFO:tasks.workunit.client.0.smithi139.stderr:pool 'test' already exists 2024-09-15T20:36:11.872 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:36:12.874 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2024-09-15T20:36:13.621 INFO:tasks.workunit.client.0.smithi139.stderr:set pool 1 size to 3 2024-09-15T20:36:13.641 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2024-09-15T20:36:14.315 INFO:tasks.workunit.client.0.smithi139.stderr:set pool 1 min_size to 2 2024-09-15T20:36:14.338 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2024-09-15T20:36:14.985 INFO:tasks.workunit.client.0.smithi139.stderr:set pool 1 pg_autoscale_mode to on 2024-09-15T20:36:15.004 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: divergent=-1 2024-09-15T20:36:15.004 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: date +%s 2024-09-15T20:36:15.006 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: start_time=1726432575 2024-09-15T20:36:15.007 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:655: TEST_divergent_3: max_duration=300 2024-09-15T20:36:15.007 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' -1 -le -1 ']' 2024-09-15T20:36:15.007 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: flush_pg_stats 2024-09-15T20:36:15.007 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:36:15.007 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:36:15.498 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:36:15.498 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:36:15.498 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:36:15.498 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:36:15.498 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:36:15.499 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:36:15.680 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836502 2024-09-15T20:36:15.680 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836502 2024-09-15T20:36:15.680 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502' 2024-09-15T20:36:15.680 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:36:15.680 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:36:15.841 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-15T20:36:15.841 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-15T20:36:15.841 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672977' 2024-09-15T20:36:15.841 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:36:15.841 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:36:15.997 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509444 2024-09-15T20:36:15.998 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509444 2024-09-15T20:36:15.998 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672977 2-64424509444' 2024-09-15T20:36:15.998 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:36:15.998 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836502 2024-09-15T20:36:15.998 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:36:15.999 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:36:15.999 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836502 2024-09-15T20:36:15.999 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:36:16.000 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836502 2024-09-15T20:36:16.000 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 21474836502 2024-09-15T20:36:16.001 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-15T20:36:16.001 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:36:16.493 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-15T20:36:16.494 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:36:16.494 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-15T20:36:16.494 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:36:16.496 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:36:16.497 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:36:16.497 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-15T20:36:16.498 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-15T20:36:16.498 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-15T20:36:16.498 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.1 seq 42949672977 2024-09-15T20:36:16.499 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:36:16.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-15T20:36:16.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:36:16.934 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509444 2024-09-15T20:36:16.934 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:36:16.935 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:36:16.936 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509444 2024-09-15T20:36:16.936 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:36:16.937 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509444 2024-09-15T20:36:16.937 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-09-15T20:36:16.938 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 64424509444 2024-09-15T20:36:16.938 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:36:17.394 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509442 -lt 64424509444 2024-09-15T20:36:17.395 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:36:18.396 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:36:18.397 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:36:19.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509444 -lt 64424509444 2024-09-15T20:36:19.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: wait_for_clean 2024-09-15T20:36:19.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:36:19.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:36:19.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:36:19.036 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:36:19.037 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:36:19.037 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:36:19.037 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:36:19.037 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:36:19.037 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:36:19.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:36:19.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:36:19.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:36:19.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:36:19.140 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:36:19.140 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:36:19.601 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:36:19.601 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:36:19.601 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:36:19.601 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:36:19.601 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:36:19.601 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:36:19.775 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836505 2024-09-15T20:36:19.775 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836505 2024-09-15T20:36:19.775 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505' 2024-09-15T20:36:19.775 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:36:19.776 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:36:19.946 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672980 2024-09-15T20:36:19.946 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672980 2024-09-15T20:36:19.946 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672980' 2024-09-15T20:36:19.946 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:36:19.947 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:36:20.098 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509447 2024-09-15T20:36:20.098 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509447 2024-09-15T20:36:20.098 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672980 2-64424509447' 2024-09-15T20:36:20.098 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:36:20.099 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836505 2024-09-15T20:36:20.099 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:36:20.100 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:36:20.100 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836505 2024-09-15T20:36:20.101 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:36:20.101 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836505 2024-09-15T20:36:20.102 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 21474836505 2024-09-15T20:36:20.102 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-15T20:36:20.102 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:36:20.605 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836505 2024-09-15T20:36:20.605 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:36:20.606 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672980 2024-09-15T20:36:20.606 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:36:20.607 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:36:20.608 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:36:20.608 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672980 2024-09-15T20:36:20.609 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672980 2024-09-15T20:36:20.609 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-15T20:36:20.609 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.1 seq 42949672980 2024-09-15T20:36:20.610 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:36:21.088 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672978 -lt 42949672980 2024-09-15T20:36:21.088 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:36:22.089 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:36:22.090 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:36:22.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672980 -lt 42949672980 2024-09-15T20:36:22.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:36:22.581 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509447 2024-09-15T20:36:22.582 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:36:22.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:36:22.584 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509447 2024-09-15T20:36:22.584 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:36:22.585 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509447 2024-09-15T20:36:22.585 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 64424509447 2024-09-15T20:36:22.586 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-15T20:36:22.586 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:36:23.067 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509447 2024-09-15T20:36:23.067 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:36:23.068 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:23.068 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:23.633 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:36:23.633 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:36:23.633 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:36:23.634 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:36:23.634 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:36:23.634 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:36:23.634 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:36:23.634 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:36:24.078 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:36:24.078 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:36:24.079 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:24.079 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:24.625 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:36:24.625 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:36:24.625 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:36:24.626 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: ceph pg dump pgs --format=json 2024-09-15T20:36:24.626 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2024-09-15T20:36:25.051 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:36:25.069 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: divergent=1 2024-09-15T20:36:25.069 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2024-09-15T20:36:25.069 INFO:tasks.workunit.client.0.smithi139.stdout:primary and soon to be divergent is 1 2024-09-15T20:36:25.069 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:665: TEST_divergent_3: ceph pg dump pgs 2024-09-15T20:36:25.466 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:36:25.466 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T20:36:12.489121+0000 0'0 31:22 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:36:11.616735+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 periodic scrub scheduled @ 2024-09-17T08:18:29.613824+0000 0 0 2024-09-15T20:36:25.467 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:36:25.467 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:36:25.467 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:36:25.482 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: date +%s 2024-09-15T20:36:25.483 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: current_time=1726432585 2024-09-15T20:36:25.484 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: expr 1726432585 - 1726432575 2024-09-15T20:36:25.486 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: elapsed_time=10 2024-09-15T20:36:25.486 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: '[' 10 -gt 300 ']' 2024-09-15T20:36:25.486 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' 1 -le -1 ']' 2024-09-15T20:36:25.487 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:675: TEST_divergent_3: local non_divergent= 2024-09-15T20:36:25.487 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-15T20:36:25.487 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 0 = 1 ']' 2024-09-15T20:36:25.487 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0' 2024-09-15T20:36:25.487 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-15T20:36:25.487 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 1 = 1 ']' 2024-09-15T20:36:25.487 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:679: TEST_divergent_3: continue 2024-09-15T20:36:25.487 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-15T20:36:25.487 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 2 = 1 ']' 2024-09-15T20:36:25.487 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0 2' 2024-09-15T20:36:25.487 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:684: TEST_divergent_3: echo 'writing initial objects' 2024-09-15T20:36:25.488 INFO:tasks.workunit.client.0.smithi139.stdout:writing initial objects 2024-09-15T20:36:25.488 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: seq 1 100 2024-09-15T20:36:25.488 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:25.488 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2024-09-15T20:36:25.645 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:25.646 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2024-09-15T20:36:25.789 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:25.789 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2024-09-15T20:36:25.957 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:25.957 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2024-09-15T20:36:26.098 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:26.098 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2024-09-15T20:36:26.257 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:26.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2024-09-15T20:36:26.427 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:26.427 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2024-09-15T20:36:26.577 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:26.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2024-09-15T20:36:26.761 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:26.761 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2024-09-15T20:36:26.928 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:26.928 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2024-09-15T20:36:27.079 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:27.079 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2024-09-15T20:36:27.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:27.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2024-09-15T20:36:27.380 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:27.380 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2024-09-15T20:36:27.532 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:27.532 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2024-09-15T20:36:27.690 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:27.690 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2024-09-15T20:36:27.841 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:27.841 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2024-09-15T20:36:28.017 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:28.017 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2024-09-15T20:36:28.189 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:28.189 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2024-09-15T20:36:28.495 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:28.495 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2024-09-15T20:36:28.697 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:28.698 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2024-09-15T20:36:28.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:28.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2024-09-15T20:36:29.148 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:29.148 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2024-09-15T20:36:29.307 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:29.307 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2024-09-15T20:36:29.465 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:29.466 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2024-09-15T20:36:29.617 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:29.617 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2024-09-15T20:36:29.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:29.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2024-09-15T20:36:29.926 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:29.927 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2024-09-15T20:36:30.100 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:30.100 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2024-09-15T20:36:31.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:31.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2024-09-15T20:36:31.437 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:31.437 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2024-09-15T20:36:31.596 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:31.596 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2024-09-15T20:36:31.805 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:31.805 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2024-09-15T20:36:31.972 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:31.973 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2024-09-15T20:36:32.124 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:32.124 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2024-09-15T20:36:32.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:32.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2024-09-15T20:36:32.459 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:32.459 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2024-09-15T20:36:32.611 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:32.611 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2024-09-15T20:36:32.778 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:32.778 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2024-09-15T20:36:32.929 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:32.929 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2024-09-15T20:36:33.071 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:33.071 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2024-09-15T20:36:33.240 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:33.240 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2024-09-15T20:36:33.416 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:33.416 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2024-09-15T20:36:33.574 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:33.575 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2024-09-15T20:36:33.809 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:33.809 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2024-09-15T20:36:33.960 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:33.960 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2024-09-15T20:36:34.186 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:34.186 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2024-09-15T20:36:35.454 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:35.454 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2024-09-15T20:36:35.679 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:35.679 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2024-09-15T20:36:35.899 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:35.899 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2024-09-15T20:36:36.065 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:36.065 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2024-09-15T20:36:36.272 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:36.272 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2024-09-15T20:36:36.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:36.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2024-09-15T20:36:36.989 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:36.989 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2024-09-15T20:36:37.579 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:37.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2024-09-15T20:36:37.794 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:37.794 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2024-09-15T20:36:37.996 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:37.996 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2024-09-15T20:36:38.231 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:38.231 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2024-09-15T20:36:38.560 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:38.560 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2024-09-15T20:36:38.946 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:38.947 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2024-09-15T20:36:39.355 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:39.355 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2024-09-15T20:36:39.675 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:39.675 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2024-09-15T20:36:39.849 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:39.850 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2024-09-15T20:36:40.001 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:40.002 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2024-09-15T20:36:40.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:40.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2024-09-15T20:36:40.597 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:40.597 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2024-09-15T20:36:40.919 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:40.919 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2024-09-15T20:36:41.075 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:41.076 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2024-09-15T20:36:41.261 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:41.261 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2024-09-15T20:36:41.529 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:41.530 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2024-09-15T20:36:41.754 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:41.754 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2024-09-15T20:36:41.957 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:41.957 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2024-09-15T20:36:42.132 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:42.132 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2024-09-15T20:36:42.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:42.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2024-09-15T20:36:42.661 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:42.661 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2024-09-15T20:36:42.921 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:42.922 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2024-09-15T20:36:43.106 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:43.106 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2024-09-15T20:36:43.292 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:43.292 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2024-09-15T20:36:43.492 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:43.492 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2024-09-15T20:36:43.685 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:43.685 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2024-09-15T20:36:43.882 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:43.882 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2024-09-15T20:36:44.046 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:44.047 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2024-09-15T20:36:44.246 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:44.246 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2024-09-15T20:36:44.684 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:44.684 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2024-09-15T20:36:45.424 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:45.424 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2024-09-15T20:36:45.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:45.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2024-09-15T20:36:45.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:45.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2024-09-15T20:36:45.967 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:45.967 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2024-09-15T20:36:46.135 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:46.135 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2024-09-15T20:36:46.471 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:46.471 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2024-09-15T20:36:46.623 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:46.623 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2024-09-15T20:36:46.789 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:46.789 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2024-09-15T20:36:46.940 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:46.940 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2024-09-15T20:36:47.091 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:47.091 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2024-09-15T20:36:47.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:47.242 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2024-09-15T20:36:47.402 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:47.402 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2024-09-15T20:36:47.577 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:47.577 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2024-09-15T20:36:47.745 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:47.745 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2024-09-15T20:36:47.896 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:47.896 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2024-09-15T20:36:48.072 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:48.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2024-09-15T20:36:48.534 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T20:36:48.534 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2024-09-15T20:36:49.004 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T20:36:49.004 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: wait_for_clean 2024-09-15T20:36:49.004 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:36:49.004 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:36:49.004 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:36:49.005 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T20:36:49.005 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:36:49.006 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:36:49.006 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:36:49.006 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:36:49.007 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:36:49.082 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T20:36:49.083 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:36:49.083 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:36:49.083 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:36:49.083 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:36:49.083 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:36:49.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:36:49.538 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:36:49.538 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:36:49.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:36:49.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:36:49.538 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:36:49.758 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836513 2024-09-15T20:36:49.758 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836513 2024-09-15T20:36:49.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513' 2024-09-15T20:36:49.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:36:49.759 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:36:49.916 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672988 2024-09-15T20:36:49.916 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672988 2024-09-15T20:36:49.917 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-42949672988' 2024-09-15T20:36:49.917 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:36:49.917 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:36:50.160 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-15T20:36:50.161 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-15T20:36:50.161 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-42949672988 2-64424509455' 2024-09-15T20:36:50.161 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:36:50.161 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836513 2024-09-15T20:36:50.161 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:36:50.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:36:50.163 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836513 2024-09-15T20:36:50.163 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:36:50.164 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836513 2024-09-15T20:36:50.164 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-09-15T20:36:50.165 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 21474836513 2024-09-15T20:36:50.165 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:36:50.754 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836513 -lt 21474836513 2024-09-15T20:36:50.754 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:36:50.754 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672988 2024-09-15T20:36:50.755 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:36:50.757 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:36:50.758 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672988 2024-09-15T20:36:50.758 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:36:50.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672988 2024-09-15T20:36:50.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-09-15T20:36:50.759 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.1 seq 42949672988 2024-09-15T20:36:50.759 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:36:51.251 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672986 -lt 42949672988 2024-09-15T20:36:51.251 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:36:52.252 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:36:52.253 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:36:52.854 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672988 -lt 42949672988 2024-09-15T20:36:52.854 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:36:52.855 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:36:52.855 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509455 2024-09-15T20:36:52.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:36:52.857 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509455 2024-09-15T20:36:52.857 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:36:52.858 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-15T20:36:52.858 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-15T20:36:52.859 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 64424509455 2024-09-15T20:36:52.859 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:36:53.322 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-15T20:36:53.323 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:36:53.323 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:53.323 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:53.910 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-15T20:36:53.910 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:36:53.910 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:36:53.911 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:36:53.911 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:36:53.911 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:36:53.911 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:36:53.911 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:36:54.361 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=30 2024-09-15T20:36:54.361 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:36:54.362 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:54.362 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:54.955 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 30 = 32 2024-09-15T20:36:54.956 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 30 '!=' -1 2024-09-15T20:36:54.956 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:36:54.956 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=30 2024-09-15T20:36:54.956 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:36:54.956 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:36:55.057 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:36:55.057 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:36:55.057 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:36:55.058 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:36:55.058 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:36:55.058 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:36:55.058 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:36:55.058 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:36:55.537 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=30 2024-09-15T20:36:55.537 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:36:55.538 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:55.538 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:56.091 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 30 = 32 2024-09-15T20:36:56.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 30 '!=' 30 2024-09-15T20:36:56.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:36:56.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:36:56.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:36:56.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:36:56.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:36:56.092 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:36:56.092 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:36:56.639 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:36:56.640 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:36:56.640 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-15T20:36:56.640 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:36:56.640 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:36:56.841 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:36:56.841 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:36:56.842 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:36:56.842 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:36:56.842 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:36:56.842 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:36:56.842 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:36:56.842 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:36:57.284 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=30 2024-09-15T20:36:57.310 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:36:57.310 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:57.310 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:57.842 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 30 = 32 2024-09-15T20:36:57.842 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 30 '!=' 30 2024-09-15T20:36:57.842 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:36:57.842 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:36:57.842 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:36:57.842 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:36:57.842 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:36:57.843 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:36:57.843 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:36:58.550 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:36:58.551 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:36:58.551 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-15T20:36:58.551 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:36:58.551 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T20:36:58.952 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:36:58.952 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:36:58.954 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:36:58.954 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:36:58.954 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:36:58.954 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:36:58.954 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:36:58.955 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:36:59.405 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=32 2024-09-15T20:36:59.405 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:36:59.406 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:36:59.406 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:36:59.967 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 = 32 2024-09-15T20:36:59.967 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:36:59.967 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:36:59.968 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: get_pg test existing_1 2024-09-15T20:36:59.968 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-15T20:36:59.968 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-15T20:36:59.968 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-15T20:36:59.968 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:37:00.452 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: local pgid=1.1b 2024-09-15T20:37:00.453 INFO:tasks.workunit.client.0.smithi139.stdout:blackholing osds 0 2 2024-09-15T20:37:00.453 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: echo 'blackholing osds 0 2' 2024-09-15T20:37:00.453 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:697: TEST_divergent_3: ceph pg dump pgs 2024-09-15T20:37:00.866 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:37:00.866 INFO:tasks.workunit.client.0.smithi139.stdout:1.1f 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:36:40.657789+0000 41'29 51:114 [1,2,0] 1 [1,2,0] 1 37'28 2024-09-15T20:36:40.657710+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T05:55:02.289067+0000 2 0 2024-09-15T20:37:00.866 INFO:tasks.workunit.client.0.smithi139.stdout:1.1e 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:36:45.584848+0000 46'29 51:113 [1,0,2] 1 [1,0,2] 1 40'28 2024-09-15T20:36:45.584734+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:51:25.929030+0000 2 0 2024-09-15T20:37:00.866 INFO:tasks.workunit.client.0.smithi139.stdout:1.1d 4 0 0 0 0 2468 0 0 19 12 19 active+clean 2024-09-15T20:36:37.494065+0000 42'31 52:129 [2,0,1] 2 [2,0,1] 2 36'29 2024-09-15T20:36:37.493943+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:47:17.221801+0000 2 0 2024-09-15T20:37:00.866 INFO:tasks.workunit.client.0.smithi139.stdout:1.b 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:36:44.549545+0000 45'29 51:114 [1,2,0] 1 [1,2,0] 1 34'28 2024-09-15T20:36:44.549410+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:11:32.721599+0000 2 0 2024-09-15T20:37:00.867 INFO:tasks.workunit.client.0.smithi139.stdout:1.a 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:36:56.804871+0000 39'29 51:96 [1,2,0] 1 [1,2,0] 1 39'29 2024-09-15T20:36:56.804751+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:44:34.935360+0000 3 0 2024-09-15T20:37:00.867 INFO:tasks.workunit.client.0.smithi139.stdout:1.9 5 0 0 0 0 3085 0 0 17 12 17 active+clean 2024-09-15T20:36:51.656456+0000 45'29 51:120 [1,2,0] 1 [1,2,0] 1 45'29 2024-09-15T20:36:51.656322+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:14:48.783354+0000 5 0 2024-09-15T20:37:00.867 INFO:tasks.workunit.client.0.smithi139.stdout:1.8 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-15T20:36:50.633724+0000 45'30 51:125 [1,2,0] 1 [1,2,0] 1 45'30 2024-09-15T20:36:50.633607+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:40:51.301964+0000 3 0 2024-09-15T20:37:00.867 INFO:tasks.workunit.client.0.smithi139.stdout:1.7 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-15T20:36:48.576984+0000 38'28 51:115 [1,2,0] 1 [1,2,0] 1 38'28 2024-09-15T20:36:48.576911+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:26:48.161399+0000 2 0 2024-09-15T20:37:00.867 INFO:tasks.workunit.client.0.smithi139.stdout:1.6 4 0 0 0 0 2468 0 0 18 12 18 active+clean 2024-09-15T20:36:46.625966+0000 42'30 51:121 [1,0,2] 1 [1,0,2] 1 42'30 2024-09-15T20:36:46.625811+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T05:09:01.852175+0000 4 0 2024-09-15T20:37:00.867 INFO:tasks.workunit.client.0.smithi139.stdout:1.5 6 0 0 0 0 3702 0 0 20 12 20 active+clean 2024-09-15T20:36:40.468243+0000 45'32 52:129 [2,0,1] 2 [2,0,1] 2 40'30 2024-09-15T20:36:40.468115+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:28:34.001242+0000 4 0 2024-09-15T20:37:00.867 INFO:tasks.workunit.client.0.smithi139.stdout:1.4 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-15T20:36:38.561622+0000 40'28 51:117 [1,0,2] 1 [1,0,2] 1 32'27 2024-09-15T20:36:38.561548+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:12:00.802143+0000 1 0 2024-09-15T20:37:00.867 INFO:tasks.workunit.client.0.smithi139.stdout:1.3 5 0 0 0 0 3085 0 0 18 12 18 active+clean 2024-09-15T20:36:54.739612+0000 47'30 51:115 [1,2,0] 1 [1,2,0] 1 47'30 2024-09-15T20:36:54.739486+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:02:26.819599+0000 5 0 2024-09-15T20:37:00.867 INFO:tasks.workunit.client.0.smithi139.stdout:1.2 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-15T20:36:44.289962+0000 42'28 51:106 [0,1,2] 0 [0,1,2] 0 42'28 2024-09-15T20:36:44.289898+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 periodic scrub scheduled @ 2024-09-17T03:21:14.945470+0000 1 0 2024-09-15T20:37:00.867 INFO:tasks.workunit.client.0.smithi139.stdout:1.c 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:36:43.561214+0000 44'29 51:118 [1,0,2] 1 [1,0,2] 1 38'28 2024-09-15T20:36:43.561129+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:57:36.145925+0000 2 0 2024-09-15T20:37:00.867 INFO:tasks.workunit.client.0.smithi139.stdout:1.1 2 0 0 0 0 1234 0 0 17 12 17 active+clean 2024-09-15T20:36:42.549934+0000 44'29 52:114 [2,0,1] 2 [2,0,1] 2 32'27 2024-09-15T20:36:42.549754+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T05:43:13.724372+0000 0 0 2024-09-15T20:37:00.867 INFO:tasks.workunit.client.0.smithi139.stdout:1.1c 2 0 0 0 0 1234 0 0 17 12 17 active+clean 2024-09-15T20:36:34.685090+0000 36'29 51:118 [1,0,2] 1 [1,0,2] 1 34'28 2024-09-15T20:36:34.684895+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:01:19.598026+0000 1 0 2024-09-15T20:37:00.868 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:36:31.575798+0000 45'29 51:118 [1,0,2] 1 [1,0,2] 1 32'27 2024-09-15T20:36:31.575682+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:53:02.339873+0000 1 0 2024-09-15T20:37:00.868 INFO:tasks.workunit.client.0.smithi139.stdout:1.d 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:36:39.548637+0000 45'29 52:115 [2,0,1] 2 [2,0,1] 2 32'27 2024-09-15T20:36:39.548519+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:24:53.675831+0000 1 0 2024-09-15T20:37:00.868 INFO:tasks.workunit.client.0.smithi139.stdout:1.e 4 0 0 0 0 2468 0 0 17 12 17 active+clean 2024-09-15T20:36:49.578187+0000 44'29 51:108 [1,0,2] 1 [1,0,2] 1 44'29 2024-09-15T20:36:49.578000+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:16:39.893426+0000 4 0 2024-09-15T20:37:00.868 INFO:tasks.workunit.client.0.smithi139.stdout:1.f 4 0 0 0 0 2468 0 0 17 12 17 active+clean 2024-09-15T20:36:39.474102+0000 41'29 51:114 [1,2,0] 1 [1,2,0] 1 34'28 2024-09-15T20:36:32.588850+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:24:03.068946+0000 3 0 2024-09-15T20:37:00.868 INFO:tasks.workunit.client.0.smithi139.stdout:1.10 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-15T20:36:39.578228+0000 45'30 51:119 [1,0,2] 1 [1,0,2] 1 38'29 2024-09-15T20:36:39.578100+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:44:46.351379+0000 2 0 2024-09-15T20:37:00.868 INFO:tasks.workunit.client.0.smithi139.stdout:1.11 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:36:36.444705+0000 37'29 52:114 [2,0,1] 2 [2,0,1] 2 36'28 2024-09-15T20:36:36.444590+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:10:36.279961+0000 2 0 2024-09-15T20:37:00.868 INFO:tasks.workunit.client.0.smithi139.stdout:1.12 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-15T20:36:37.226341+0000 36'28 51:123 [0,1,2] 0 [0,1,2] 0 32'27 2024-09-15T20:36:35.623615+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:15:26.873396+0000 0 0 2024-09-15T20:37:00.868 INFO:tasks.workunit.client.0.smithi139.stdout:1.13 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-15T20:36:55.766132+0000 45'30 51:115 [1,2,0] 1 [1,2,0] 1 45'30 2024-09-15T20:36:55.766014+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 periodic scrub scheduled @ 2024-09-17T00:51:03.799106+0000 3 0 2024-09-15T20:37:00.868 INFO:tasks.workunit.client.0.smithi139.stdout:1.14 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:36:37.554158+0000 45'29 51:118 [1,0,2] 1 [1,0,2] 1 32'27 2024-09-15T20:36:37.554032+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T22:25:53.857258+0000 1 0 2024-09-15T20:37:00.868 INFO:tasks.workunit.client.0.smithi139.stdout:1.15 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:36:41.507018+0000 41'29 52:126 [2,0,1] 2 [2,0,1] 2 41'29 2024-09-15T20:36:41.506894+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T22:16:35.452398+0000 3 0 2024-09-15T20:37:00.869 INFO:tasks.workunit.client.0.smithi139.stdout:1.16 4 0 0 0 0 2468 0 0 19 12 19 active+clean 2024-09-15T20:36:47.576707+0000 41'31 51:131 [1,0,2] 1 [1,0,2] 1 41'31 2024-09-15T20:36:47.576518+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:49:29.884446+0000 4 0 2024-09-15T20:37:00.869 INFO:tasks.workunit.client.0.smithi139.stdout:1.17 1 0 0 0 0 617 0 0 15 12 15 active+clean 2024-09-15T20:36:42.639801+0000 32'27 51:112 [1,2,0] 1 [1,2,0] 1 32'27 2024-09-15T20:36:42.639678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:25:53.980640+0000 1 0 2024-09-15T20:37:00.869 INFO:tasks.workunit.client.0.smithi139.stdout:1.18 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-15T20:36:49.055223+0000 45'30 51:125 [1,2,0] 1 [1,2,0] 1 34'28 2024-09-15T20:36:36.622678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:32:38.831691+0000 1 0 2024-09-15T20:37:00.869 INFO:tasks.workunit.client.0.smithi139.stdout:1.19 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-15T20:36:52.759876+0000 34'28 51:89 [1,2,0] 1 [1,2,0] 1 34'28 2024-09-15T20:36:52.759747+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:46:33.047111+0000 2 0 2024-09-15T20:37:00.869 INFO:tasks.workunit.client.0.smithi139.stdout:1.1a 4 0 0 0 0 2468 0 0 18 12 18 active+clean 2024-09-15T20:36:53.716843+0000 44'30 51:119 [1,2,0] 1 [1,2,0] 1 44'30 2024-09-15T20:36:53.716793+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:31:39.811226+0000 4 0 2024-09-15T20:37:00.869 INFO:tasks.workunit.client.0.smithi139.stdout:1.1b 5 0 0 0 0 3085 0 0 19 12 19 active+clean 2024-09-15T20:36:41.658454+0000 45'31 51:123 [1,2,0] 1 [1,2,0] 1 41'30 2024-09-15T20:36:41.658323+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:55:22.611960+0000 4 0 2024-09-15T20:37:00.869 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:37:00.869 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:37:00.869 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:37:00.883 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-09-15T20:37:00.884 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.0 2024-09-15T20:37:00.884 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T20:37:00.884 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T20:37:00.884 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:37:00.884 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:37:00.885 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:37:00.885 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.0.asok 2024-09-15T20:37:00.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-09-15T20:37:00.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-15T20:37:00.981 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:37:00.981 INFO:tasks.workunit.client.0.smithi139.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-15T20:37:00.981 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:37:00.992 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-09-15T20:37:00.993 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.2 2024-09-15T20:37:00.994 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-15T20:37:00.994 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-15T20:37:00.994 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:37:00.994 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:37:00.994 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:37:00.994 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.2.asok 2024-09-15T20:37:00.995 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-09-15T20:37:00.995 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-15T20:37:01.103 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:37:01.103 INFO:tasks.workunit.client.0.smithi139.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-15T20:37:01.103 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:37:01.115 INFO:tasks.workunit.client.0.smithi139.stdout:create new divergent objects 2024-09-15T20:37:01.115 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: echo 'create new divergent objects' 2024-09-15T20:37:01.116 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: seq 1 2 2024-09-15T20:37:01.118 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-15T20:37:01.118 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-15T20:37:01.119 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_1 2024-09-15T20:37:01.119 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_2 2024-09-15T20:37:01.119 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:710: TEST_divergent_3: echo 'writing divergent objects' 2024-09-15T20:37:01.119 INFO:tasks.workunit.client.0.smithi139.stdout:writing divergent objects 2024-09-15T20:37:01.119 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: seq 1 2 2024-09-15T20:37:01.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-15T20:37:01.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-15T20:37:01.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2024-09-15T20:37:01.121 INFO:tasks.workunit.client.0.smithi139.stdout:remove divergent objects 2024-09-15T20:37:01.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: echo 'remove divergent objects' 2024-09-15T20:37:01.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2024-09-15T20:37:01.122 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: seq 1 2 2024-09-15T20:37:01.123 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-15T20:37:01.124 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 1 + 2 2024-09-15T20:37:01.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=3 2024-09-15T20:37:01.126 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-15T20:37:01.126 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_3 2024-09-15T20:37:01.126 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 2 + 2 2024-09-15T20:37:01.127 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=4 2024-09-15T20:37:01.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:722: TEST_divergent_3: sleep 10 2024-09-15T20:37:01.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_4 2024-09-15T20:37:11.130 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:723: TEST_divergent_3: killall -9 rados 2024-09-15T20:37:11.136 INFO:tasks.workunit.client.0.smithi139.stderr:rados: no process found 2024-09-15T20:37:11.136 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: echo 'killing all the osds' 2024-09-15T20:37:11.136 INFO:tasks.workunit.client.0.smithi139.stdout:killing all the osds 2024-09-15T20:37:11.136 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:727: TEST_divergent_3: ceph pg dump pgs 2024-09-15T20:37:11.566 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:37:11.566 INFO:tasks.workunit.client.0.smithi139.stdout:1.1f 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:37:07.497552+0000 41'29 61:109 [2,0,1] 2 [2,0,1] 2 41'29 2024-09-15T20:37:07.497422+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:31:59.161535+0000 3 0 2024-09-15T20:37:11.567 INFO:tasks.workunit.client.0.smithi139.stdout:1.1e 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:37:04.487289+0000 46'29 60:142 [1,2,0] 1 [1,2,0] 1 40'28 2024-09-15T20:36:45.584734+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:36:05.680759+0000 2 0 2024-09-15T20:37:11.567 INFO:tasks.workunit.client.0.smithi139.stdout:1.1d 4 0 0 0 0 2468 0 0 19 12 19 active+clean 2024-09-15T20:37:02.447402+0000 42'31 61:158 [2,1,0] 2 [2,1,0] 2 36'29 2024-09-15T20:36:37.493943+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:11:24.363141+0000 2 0 2024-09-15T20:37:11.567 INFO:tasks.workunit.client.0.smithi139.stdout:1.b 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:36:58.872433+0000 45'29 60:143 [1,0,2] 1 [1,0,2] 1 34'28 2024-09-15T20:36:44.549410+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:57:00.676079+0000 2 0 2024-09-15T20:37:11.567 INFO:tasks.workunit.client.0.smithi139.stdout:1.a 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:36:56.804871+0000 39'29 60:119 [1,2,0] 1 [1,2,0] 1 39'29 2024-09-15T20:36:56.804751+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:50:08.564619+0000 3 0 2024-09-15T20:37:11.567 INFO:tasks.workunit.client.0.smithi139.stdout:1.9 5 0 0 0 0 3435 0 0 18 12 18 active+clean 2024-09-15T20:36:51.656456+0000 54'30 60:144 [1,2,0] 1 [1,2,0] 1 45'29 2024-09-15T20:36:51.656322+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:04:13.954057+0000 5 0 2024-09-15T20:37:11.567 INFO:tasks.workunit.client.0.smithi139.stdout:1.8 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-15T20:36:50.633724+0000 45'30 60:148 [1,2,0] 1 [1,2,0] 1 45'30 2024-09-15T20:36:50.633607+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:49:41.051485+0000 3 0 2024-09-15T20:37:11.567 INFO:tasks.workunit.client.0.smithi139.stdout:1.7 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-15T20:36:48.576984+0000 38'28 60:138 [1,2,0] 1 [1,2,0] 1 38'28 2024-09-15T20:36:48.576911+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T23:12:12.111458+0000 2 0 2024-09-15T20:37:11.567 INFO:tasks.workunit.client.0.smithi139.stdout:1.6 4 0 0 0 0 2468 0 0 18 12 18 active+clean 2024-09-15T20:36:46.625966+0000 42'30 60:144 [1,0,2] 1 [1,0,2] 1 42'30 2024-09-15T20:36:46.625811+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:55:07.763260+0000 4 0 2024-09-15T20:37:11.567 INFO:tasks.workunit.client.0.smithi139.stdout:1.5 6 0 0 0 0 3702 0 0 20 12 20 active+clean 2024-09-15T20:36:40.468243+0000 45'32 61:152 [2,0,1] 2 [2,0,1] 2 40'30 2024-09-15T20:36:40.468115+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T23:43:15.196718+0000 4 0 2024-09-15T20:37:11.567 INFO:tasks.workunit.client.0.smithi139.stdout:1.4 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-15T20:36:38.561622+0000 40'28 60:140 [1,0,2] 1 [1,0,2] 1 32'27 2024-09-15T20:36:38.561548+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T23:29:53.224343+0000 1 0 2024-09-15T20:37:11.567 INFO:tasks.workunit.client.0.smithi139.stdout:1.3 5 0 0 0 0 3085 0 0 18 12 18 active+clean 2024-09-15T20:36:54.739612+0000 47'30 60:138 [1,2,0] 1 [1,2,0] 1 47'30 2024-09-15T20:36:54.739486+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:35:29.352480+0000 5 0 2024-09-15T20:37:11.567 INFO:tasks.workunit.client.0.smithi139.stdout:1.2 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-15T20:36:44.289962+0000 42'28 59:127 [0,1,2] 0 [0,1,2] 0 42'28 2024-09-15T20:36:44.289898+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 periodic scrub scheduled @ 2024-09-17T03:23:17.186026+0000 1 0 2024-09-15T20:37:11.567 INFO:tasks.workunit.client.0.smithi139.stdout:1.c 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:36:43.561214+0000 44'29 60:141 [1,0,2] 1 [1,0,2] 1 38'28 2024-09-15T20:36:43.561129+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:17:43.817781+0000 2 0 2024-09-15T20:37:11.568 INFO:tasks.workunit.client.0.smithi139.stdout:1.1 2 0 0 0 0 1234 0 0 17 12 17 active+clean 2024-09-15T20:36:42.549934+0000 44'29 61:137 [2,0,1] 2 [2,0,1] 2 32'27 2024-09-15T20:36:42.549754+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T23:02:37.592554+0000 0 0 2024-09-15T20:37:11.568 INFO:tasks.workunit.client.0.smithi139.stdout:1.1c 2 0 0 0 0 1234 0 0 17 12 17 active+clean 2024-09-15T20:36:34.685090+0000 36'29 60:141 [1,0,2] 1 [1,0,2] 1 34'28 2024-09-15T20:36:34.684895+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:40:03.128114+0000 1 0 2024-09-15T20:37:11.568 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:36:31.575798+0000 45'29 60:141 [1,0,2] 1 [1,0,2] 1 32'27 2024-09-15T20:36:31.575682+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:59:51.834937+0000 1 0 2024-09-15T20:37:11.568 INFO:tasks.workunit.client.0.smithi139.stdout:1.d 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:37:02.447855+0000 45'29 61:144 [2,1,0] 2 [2,1,0] 2 32'27 2024-09-15T20:36:39.548519+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:18:17.680746+0000 1 0 2024-09-15T20:37:11.568 INFO:tasks.workunit.client.0.smithi139.stdout:1.e 4 0 0 0 0 1851 0 0 19 12 19 active+clean 2024-09-15T20:37:04.572148+0000 54'31 60:139 [1,2,0] 1 [1,2,0] 1 44'29 2024-09-15T20:36:49.578000+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:44:28.594037+0000 4 0 2024-09-15T20:37:11.568 INFO:tasks.workunit.client.0.smithi139.stdout:1.f 4 0 0 0 0 2468 0 0 17 12 17 active+clean 2024-09-15T20:37:06.664711+0000 41'29 61:101 [2,0,1] 2 [2,0,1] 2 34'28 2024-09-15T20:36:32.588850+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 queued for scrub 3 0 2024-09-15T20:37:11.568 INFO:tasks.workunit.client.0.smithi139.stdout:1.10 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-15T20:36:39.578228+0000 45'30 60:142 [1,0,2] 1 [1,0,2] 1 38'29 2024-09-15T20:36:39.578100+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:43:25.712842+0000 2 0 2024-09-15T20:37:11.568 INFO:tasks.workunit.client.0.smithi139.stdout:1.11 2 0 0 0 0 1234 0 0 18 12 18 active+clean 2024-09-15T20:36:36.444705+0000 54'30 61:138 [2,0,1] 2 [2,0,1] 2 36'28 2024-09-15T20:36:36.444590+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:33:17.040483+0000 2 0 2024-09-15T20:37:11.568 INFO:tasks.workunit.client.0.smithi139.stdout:1.12 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-15T20:36:37.226341+0000 36'28 59:144 [0,1,2] 0 [0,1,2] 0 32'27 2024-09-15T20:36:35.623615+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:26:25.094711+0000 0 0 2024-09-15T20:37:11.568 INFO:tasks.workunit.client.0.smithi139.stdout:1.13 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-15T20:36:55.766132+0000 45'30 60:138 [1,2,0] 1 [1,2,0] 1 45'30 2024-09-15T20:36:55.766014+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 periodic scrub scheduled @ 2024-09-17T04:21:20.570509+0000 3 0 2024-09-15T20:37:11.568 INFO:tasks.workunit.client.0.smithi139.stdout:1.14 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:36:37.554158+0000 45'29 60:141 [1,0,2] 1 [1,0,2] 1 32'27 2024-09-15T20:36:37.554032+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:54:11.824454+0000 1 0 2024-09-15T20:37:11.568 INFO:tasks.workunit.client.0.smithi139.stdout:1.15 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:36:41.507018+0000 41'29 61:149 [2,0,1] 2 [2,0,1] 2 41'29 2024-09-15T20:36:41.506894+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:50:36.842486+0000 3 0 2024-09-15T20:37:11.569 INFO:tasks.workunit.client.0.smithi139.stdout:1.16 4 0 0 0 0 2468 0 0 19 12 19 active+clean 2024-09-15T20:36:47.576707+0000 41'31 60:154 [1,0,2] 1 [1,0,2] 1 41'31 2024-09-15T20:36:47.576518+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:03:53.353281+0000 4 0 2024-09-15T20:37:11.569 INFO:tasks.workunit.client.0.smithi139.stdout:1.17 1 0 0 0 0 617 0 0 15 12 15 active+clean 2024-09-15T20:36:42.639801+0000 32'27 60:135 [1,2,0] 1 [1,2,0] 1 32'27 2024-09-15T20:36:42.639678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:24:03.913853+0000 1 0 2024-09-15T20:37:11.569 INFO:tasks.workunit.client.0.smithi139.stdout:1.18 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-15T20:36:49.055223+0000 45'30 60:148 [1,2,0] 1 [1,2,0] 1 34'28 2024-09-15T20:36:36.622678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T05:00:08.064376+0000 1 0 2024-09-15T20:37:11.569 INFO:tasks.workunit.client.0.smithi139.stdout:1.19 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-15T20:36:52.759876+0000 34'28 60:112 [1,2,0] 1 [1,2,0] 1 34'28 2024-09-15T20:36:52.759747+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:37:58.883870+0000 2 0 2024-09-15T20:37:11.569 INFO:tasks.workunit.client.0.smithi139.stdout:1.1a 4 0 0 0 0 2468 0 0 18 12 18 active+clean 2024-09-15T20:36:53.716843+0000 44'30 60:142 [1,2,0] 1 [1,2,0] 1 44'30 2024-09-15T20:36:53.716793+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T22:59:05.662754+0000 4 0 2024-09-15T20:37:11.569 INFO:tasks.workunit.client.0.smithi139.stdout:1.1b 6 0 0 0 0 3435 0 0 21 12 21 active+clean 2024-09-15T20:36:58.872336+0000 54'33 60:154 [1,0,2] 1 [1,0,2] 1 41'30 2024-09-15T20:36:41.658323+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:12:56.414997+0000 4 0 2024-09-15T20:37:11.569 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:37:11.569 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:37:11.569 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:37:11.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2024-09-15T20:37:11.583 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:37:11.583 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:37:11.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:37:11.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:37:11.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:37:11.893 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:37:11.893 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-15T20:37:11.893 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.0 2024-09-15T20:37:12.460 INFO:tasks.workunit.client.0.smithi139.stderr:osd.0 is already down. 2024-09-15T20:37:12.480 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-15T20:37:12.481 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.1 2024-09-15T20:37:13.034 INFO:tasks.workunit.client.0.smithi139.stderr:osd.1 is already down. 2024-09-15T20:37:13.051 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-15T20:37:13.052 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.2 2024-09-15T20:37:13.602 INFO:tasks.workunit.client.0.smithi139.stderr:osd.2 is already down. 2024-09-15T20:37:13.618 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-09-15T20:37:13.618 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.0 2024-09-15T20:37:14.395 INFO:tasks.workunit.client.0.smithi139.stderr:osd.0 is already out. 2024-09-15T20:37:14.412 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-09-15T20:37:14.412 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.2 2024-09-15T20:37:14.966 INFO:tasks.workunit.client.0.smithi139.stderr:osd.2 is already out. 2024-09-15T20:37:14.983 INFO:tasks.workunit.client.0.smithi139.stdout:bringing up non_divergent 0 2 2024-09-15T20:37:14.983 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2024-09-15T20:37:14.983 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2024-09-15T20:37:15.388 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:37:15.388 INFO:tasks.workunit.client.0.smithi139.stdout:1.1f 3 0 0 0 0 1851 0 0 17 12 17 stale+active+clean 2024-09-15T20:37:07.497552+0000 41'29 61:109 [2,0,1] 2 [2,0,1] 2 41'29 2024-09-15T20:37:07.497422+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:31:59.161535+0000 3 0 2024-09-15T20:37:15.388 INFO:tasks.workunit.client.0.smithi139.stdout:1.1e 3 0 0 0 0 1851 0 0 17 12 17 stale+active+clean 2024-09-15T20:37:04.487289+0000 46'29 60:142 [1,2,0] 1 [1,2,0] 1 40'28 2024-09-15T20:36:45.584734+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:36:05.680759+0000 2 0 2024-09-15T20:37:15.388 INFO:tasks.workunit.client.0.smithi139.stdout:1.1d 4 0 0 0 0 2468 0 0 19 12 19 stale+active+clean 2024-09-15T20:37:02.447402+0000 42'31 61:158 [2,1,0] 2 [2,1,0] 2 36'29 2024-09-15T20:36:37.493943+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:11:24.363141+0000 2 0 2024-09-15T20:37:15.388 INFO:tasks.workunit.client.0.smithi139.stdout:1.b 3 0 0 0 0 1851 0 0 17 12 17 stale+active+clean 2024-09-15T20:36:58.872433+0000 45'29 60:143 [1,0,2] 1 [1,0,2] 1 34'28 2024-09-15T20:36:44.549410+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:57:00.676079+0000 2 0 2024-09-15T20:37:15.388 INFO:tasks.workunit.client.0.smithi139.stdout:1.a 3 0 0 0 0 1851 0 0 17 12 17 stale+active+clean 2024-09-15T20:36:56.804871+0000 39'29 60:119 [1,2,0] 1 [1,2,0] 1 39'29 2024-09-15T20:36:56.804751+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:50:08.564619+0000 3 0 2024-09-15T20:37:15.388 INFO:tasks.workunit.client.0.smithi139.stdout:1.9 5 0 0 0 0 3435 0 0 18 12 18 stale+active+clean 2024-09-15T20:36:51.656456+0000 54'30 60:144 [1,2,0] 1 [1,2,0] 1 45'29 2024-09-15T20:36:51.656322+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:04:13.954057+0000 5 0 2024-09-15T20:37:15.388 INFO:tasks.workunit.client.0.smithi139.stdout:1.8 3 0 0 0 0 1851 0 0 18 12 18 stale+active+clean 2024-09-15T20:36:50.633724+0000 45'30 60:148 [1,2,0] 1 [1,2,0] 1 45'30 2024-09-15T20:36:50.633607+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:49:41.051485+0000 3 0 2024-09-15T20:37:15.388 INFO:tasks.workunit.client.0.smithi139.stdout:1.7 2 0 0 0 0 1234 0 0 16 12 16 stale+active+clean 2024-09-15T20:36:48.576984+0000 38'28 60:138 [1,2,0] 1 [1,2,0] 1 38'28 2024-09-15T20:36:48.576911+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T23:12:12.111458+0000 2 0 2024-09-15T20:37:15.389 INFO:tasks.workunit.client.0.smithi139.stdout:1.6 4 0 0 0 0 2468 0 0 18 12 18 stale+active+clean 2024-09-15T20:36:46.625966+0000 42'30 60:144 [1,0,2] 1 [1,0,2] 1 42'30 2024-09-15T20:36:46.625811+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:55:07.763260+0000 4 0 2024-09-15T20:37:15.389 INFO:tasks.workunit.client.0.smithi139.stdout:1.5 6 0 0 0 0 3702 0 0 20 12 20 stale+active+clean 2024-09-15T20:36:40.468243+0000 45'32 61:152 [2,0,1] 2 [2,0,1] 2 40'30 2024-09-15T20:36:40.468115+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T23:43:15.196718+0000 4 0 2024-09-15T20:37:15.389 INFO:tasks.workunit.client.0.smithi139.stdout:1.4 2 0 0 0 0 1234 0 0 16 12 16 stale+active+clean 2024-09-15T20:36:38.561622+0000 40'28 60:140 [1,0,2] 1 [1,0,2] 1 32'27 2024-09-15T20:36:38.561548+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T23:29:53.224343+0000 1 0 2024-09-15T20:37:15.389 INFO:tasks.workunit.client.0.smithi139.stdout:1.3 5 0 0 0 0 3085 0 0 18 12 18 stale+active+clean 2024-09-15T20:36:54.739612+0000 47'30 60:138 [1,2,0] 1 [1,2,0] 1 47'30 2024-09-15T20:36:54.739486+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:35:29.352480+0000 5 0 2024-09-15T20:37:15.389 INFO:tasks.workunit.client.0.smithi139.stdout:1.2 1 0 0 0 0 617 0 0 16 12 16 stale+active+clean 2024-09-15T20:36:44.289962+0000 42'28 64:139 [0,1,2] 0 [0,1,2] 0 42'28 2024-09-15T20:36:44.289898+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 periodic scrub scheduled @ 2024-09-17T05:40:28.409639+0000 1 0 2024-09-15T20:37:15.389 INFO:tasks.workunit.client.0.smithi139.stdout:1.c 3 0 0 0 0 1851 0 0 17 12 17 stale+active+clean 2024-09-15T20:36:43.561214+0000 44'29 60:141 [1,0,2] 1 [1,0,2] 1 38'28 2024-09-15T20:36:43.561129+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:17:43.817781+0000 2 0 2024-09-15T20:37:15.389 INFO:tasks.workunit.client.0.smithi139.stdout:1.1 2 0 0 0 0 1234 0 0 17 12 17 stale+active+clean 2024-09-15T20:36:42.549934+0000 44'29 61:137 [2,0,1] 2 [2,0,1] 2 32'27 2024-09-15T20:36:42.549754+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T23:02:37.592554+0000 0 0 2024-09-15T20:37:15.389 INFO:tasks.workunit.client.0.smithi139.stdout:1.1c 2 0 0 0 0 1234 0 0 17 12 17 stale+active+clean 2024-09-15T20:36:34.685090+0000 36'29 60:141 [1,0,2] 1 [1,0,2] 1 34'28 2024-09-15T20:36:34.684895+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:40:03.128114+0000 1 0 2024-09-15T20:37:15.389 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 3 0 0 0 0 1851 0 0 17 12 17 stale+active+clean 2024-09-15T20:36:31.575798+0000 45'29 60:141 [1,0,2] 1 [1,0,2] 1 32'27 2024-09-15T20:36:31.575682+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:59:51.834937+0000 1 0 2024-09-15T20:37:15.389 INFO:tasks.workunit.client.0.smithi139.stdout:1.d 3 0 0 0 0 1851 0 0 17 12 17 stale+active+clean 2024-09-15T20:37:02.447855+0000 45'29 61:144 [2,1,0] 2 [2,1,0] 2 32'27 2024-09-15T20:36:39.548519+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:18:17.680746+0000 1 0 2024-09-15T20:37:15.389 INFO:tasks.workunit.client.0.smithi139.stdout:1.e 4 0 0 0 0 1851 0 0 19 12 19 stale+active+clean 2024-09-15T20:37:04.572148+0000 54'31 60:139 [1,2,0] 1 [1,2,0] 1 44'29 2024-09-15T20:36:49.578000+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:44:28.594037+0000 4 0 2024-09-15T20:37:15.389 INFO:tasks.workunit.client.0.smithi139.stdout:1.f 4 0 0 0 0 2468 0 0 17 12 17 stale+active+clean 2024-09-15T20:37:06.664711+0000 41'29 61:101 [2,0,1] 2 [2,0,1] 2 34'28 2024-09-15T20:36:32.588850+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 queued for scrub 3 0 2024-09-15T20:37:15.389 INFO:tasks.workunit.client.0.smithi139.stdout:1.10 3 0 0 0 0 1851 0 0 18 12 18 stale+active+clean 2024-09-15T20:37:08.503531+0000 45'30 64:120 [0,2,1] 0 [0,2,1] 0 38'29 2024-09-15T20:36:39.578100+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:11:41.509220+0000 2 0 2024-09-15T20:37:15.390 INFO:tasks.workunit.client.0.smithi139.stdout:1.11 2 0 0 0 0 1234 0 0 18 12 18 stale+active+clean 2024-09-15T20:37:10.503835+0000 54'30 64:131 [0,1,2] 0 [0,1,2] 0 36'28 2024-09-15T20:36:36.444590+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:22:41.689976+0000 2 0 2024-09-15T20:37:15.390 INFO:tasks.workunit.client.0.smithi139.stdout:1.12 1 0 0 0 0 617 0 0 16 12 16 stale+active+clean 2024-09-15T20:36:37.226341+0000 36'28 64:156 [0,1,2] 0 [0,1,2] 0 32'27 2024-09-15T20:36:35.623615+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T05:17:05.990867+0000 0 0 2024-09-15T20:37:15.390 INFO:tasks.workunit.client.0.smithi139.stdout:1.13 3 0 0 0 0 1851 0 0 18 12 18 stale+active+clean 2024-09-15T20:36:55.766132+0000 45'30 60:138 [1,2,0] 1 [1,2,0] 1 45'30 2024-09-15T20:36:55.766014+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 periodic scrub scheduled @ 2024-09-17T04:21:20.570509+0000 3 0 2024-09-15T20:37:15.390 INFO:tasks.workunit.client.0.smithi139.stdout:1.14 3 0 0 0 0 1851 0 0 17 12 17 stale+active+clean 2024-09-15T20:36:37.554158+0000 45'29 60:141 [1,0,2] 1 [1,0,2] 1 32'27 2024-09-15T20:36:37.554032+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:54:11.824454+0000 1 0 2024-09-15T20:37:15.390 INFO:tasks.workunit.client.0.smithi139.stdout:1.15 3 0 0 0 0 1851 0 0 17 12 17 stale+active+clean 2024-09-15T20:36:41.507018+0000 41'29 61:149 [2,0,1] 2 [2,0,1] 2 41'29 2024-09-15T20:36:41.506894+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:50:36.842486+0000 3 0 2024-09-15T20:37:15.390 INFO:tasks.workunit.client.0.smithi139.stdout:1.16 4 0 0 0 0 2468 0 0 19 12 19 stale+active+clean 2024-09-15T20:36:47.576707+0000 41'31 60:154 [1,0,2] 1 [1,0,2] 1 41'31 2024-09-15T20:36:47.576518+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:03:53.353281+0000 4 0 2024-09-15T20:37:15.390 INFO:tasks.workunit.client.0.smithi139.stdout:1.17 1 0 0 0 0 617 0 0 15 12 15 stale+active+clean 2024-09-15T20:36:42.639801+0000 32'27 60:135 [1,2,0] 1 [1,2,0] 1 32'27 2024-09-15T20:36:42.639678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:24:03.913853+0000 1 0 2024-09-15T20:37:15.390 INFO:tasks.workunit.client.0.smithi139.stdout:1.18 3 0 0 0 0 1851 0 0 18 12 18 stale+active+clean 2024-09-15T20:36:49.055223+0000 45'30 60:148 [1,2,0] 1 [1,2,0] 1 34'28 2024-09-15T20:36:36.622678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T05:00:08.064376+0000 1 0 2024-09-15T20:37:15.390 INFO:tasks.workunit.client.0.smithi139.stdout:1.19 2 0 0 0 0 1234 0 0 16 12 16 stale+active+clean 2024-09-15T20:36:52.759876+0000 34'28 60:112 [1,2,0] 1 [1,2,0] 1 34'28 2024-09-15T20:36:52.759747+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:37:58.883870+0000 2 0 2024-09-15T20:37:15.390 INFO:tasks.workunit.client.0.smithi139.stdout:1.1a 4 0 0 0 0 2468 0 0 18 12 18 stale+active+clean 2024-09-15T20:36:53.716843+0000 44'30 60:142 [1,2,0] 1 [1,2,0] 1 44'30 2024-09-15T20:36:53.716793+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T22:59:05.662754+0000 4 0 2024-09-15T20:37:15.390 INFO:tasks.workunit.client.0.smithi139.stdout:1.1b 6 0 0 0 0 3435 0 0 21 12 21 stale+active+clean 2024-09-15T20:36:58.872336+0000 54'33 60:154 [1,0,2] 1 [1,0,2] 1 41'30 2024-09-15T20:36:41.658323+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:12:56.414997+0000 4 0 2024-09-15T20:37:15.390 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:37:15.390 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:37:15.391 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:37:15.405 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-09-15T20:37:15.405 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 0 2024-09-15T20:37:15.405 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-15T20:37:15.405 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:37:15.405 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:37:15.405 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:37:15.406 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-15T20:37:15.406 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:37:15.406 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:37:15.406 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:37:15.406 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:37:15.406 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-15T20:37:15.406 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-15T20:37:15.406 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:37:15.406 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:37:15.406 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:37:15.406 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:37:15.406 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:37:15.406 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:37:15.408 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:37:15.408 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:37:15.408 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:37:15.408 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:37:15.408 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:37:15.408 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:37:15.408 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:37:15.409 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:37:15.409 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:37:15.409 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:37:15.409 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:37:15.409 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:37:15.409 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:37:15.409 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:37:15.409 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-15T20:37:15.409 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:37:15.409 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T20:37:15.410 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:37:15.410 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-15T20:37:15.410 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:37:15.410 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:37:15.410 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:37:15.411 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:37:15.441 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:37:15.440+0000 7f0ce3e6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:37:15.443 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:37:15.444+0000 7f0ce3e6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:37:15.444 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:37:15.444+0000 7f0ce3e6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:37:15.893 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:37:15.893 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:37:15.893 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:37:15.894 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:37:15.894 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:37:15.894 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:15.894 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:37:15.894 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:37:15.894 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:15.894 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:16.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:16.712 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:37:16.712+0000 7f0ce3e6d7c0 -1 Falling back to public interface 2024-09-15T20:37:17.446 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:17.446 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:17.446 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:37:17.447 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:37:17.447 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:17.447 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:17.928 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:18.929 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:18.929 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:37:18.930 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:18.930 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:37:18.930 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:18.930 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:19.358 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:20.007 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:37:20.008+0000 7f0ce3e6d7c0 -1 osd.0 54 log_to_monitors true 2024-09-15T20:37:20.359 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:20.359 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:20.359 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:37:20.360 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:37:20.360 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:20.360 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:37:20.971 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up out weight 0 up_from 76 up_thru 63 down_at 66 last_clean_interval [5,54) [v2:127.0.0.1:6802/3684664553,v1:127.0.0.1:6803/3684664553] [v2:127.0.0.1:6804/3684664553,v1:127.0.0.1:6805/3684664553] exists,up 8ceaad64-e87a-460e-8e98-3caf0f7eff12 2024-09-15T20:37:20.974 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:37:20.974 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:37:20.974 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:37:20.974 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-09-15T20:37:20.974 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 2 2024-09-15T20:37:20.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-15T20:37:20.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:37:20.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T20:37:20.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:37:20.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-15T20:37:20.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:37:20.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:37:20.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:37:20.976 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:37:20.976 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-15T20:37:20.976 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-15T20:37:20.976 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:37:20.976 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:37:20.976 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:37:20.977 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:37:20.977 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:37:20.977 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:37:20.977 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:37:20.977 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:37:20.977 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:37:20.977 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:37:20.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:37:20.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:37:20.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:37:20.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:37:20.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:37:20.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:37:20.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:37:20.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:37:20.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:37:20.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:37:20.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-15T20:37:20.978 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T20:37:20.979 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.2 2024-09-15T20:37:20.979 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:37:20.979 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-15T20:37:20.979 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T20:37:20.980 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:37:20.980 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:37:20.980 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:37:21.014 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:37:21.016+0000 7f2a9e81b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:37:21.020 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:37:21.020+0000 7f2a9e81b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:37:21.021 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:37:21.024+0000 7f2a9e81b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:37:21.480 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T20:37:21.480 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:37:21.480 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:37:21.480 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:37:21.481 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:37:21.481 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:21.481 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:37:21.481 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:37:21.481 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:21.481 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:37:21.996 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:22.288 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:37:22.288+0000 7f2a9e81b7c0 -1 Falling back to public interface 2024-09-15T20:37:22.999 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:22.999 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:22.999 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:37:23.000 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:37:23.000 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:23.001 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:37:23.488 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:24.490 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:24.490 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:24.490 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:37:24.491 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:37:24.491 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:24.491 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:37:24.957 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:37:25.677 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:37:25.676+0000 7f2a9e81b7c0 -1 osd.2 53 log_to_monitors true 2024-09-15T20:37:25.958 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:37:25.959 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:37:25.959 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:37:25.959 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:37:25.960 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:37:25.960 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:37:26.514 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up out weight 0 up_from 80 up_thru 59 down_at 68 last_clean_interval [15,53) [v2:127.0.0.1:6810/3676612316,v1:127.0.0.1:6811/3676612316] [v2:127.0.0.1:6812/3676612316,v1:127.0.0.1:6813/3676612316] exists,up 5347fe2a-12cd-44fe-9cd9-0dbc2bc7ebb8 2024-09-15T20:37:26.514 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:37:26.514 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:37:26.515 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:37:26.515 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-09-15T20:37:26.515 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.0 2024-09-15T20:37:27.066 INFO:tasks.workunit.client.0.smithi139.stderr:osd.0 is already in. 2024-09-15T20:37:27.083 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-09-15T20:37:27.083 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.2 2024-09-15T20:37:27.847 INFO:tasks.workunit.client.0.smithi139.stderr:osd.2 is already in. 2024-09-15T20:37:27.868 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T20:37:27.868 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: wait_for_clean 2024-09-15T20:37:27.868 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:37:27.868 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:37:27.868 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:37:27.869 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T20:37:27.869 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:37:27.869 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:37:27.870 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:37:27.870 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:37:27.870 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:37:27.935 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T20:37:27.935 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:37:27.935 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:37:27.935 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:37:27.936 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:37:27.936 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:37:28.593 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:37:28.593 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:37:28.593 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:37:28.594 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:37:28.594 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:37:28.594 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:37:28.818 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=326417514500 2024-09-15T20:37:28.818 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 326417514500 2024-09-15T20:37:28.818 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-326417514500' 2024-09-15T20:37:28.818 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:37:28.818 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:37:28.949 INFO:tasks.workunit.client.0.smithi139.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-15T20:37:28.951 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-15T20:37:28.951 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-15T20:37:28.951 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-15T20:37:28.951 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:37:28.951 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:37:29.165 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383683 2024-09-15T20:37:29.165 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383683 2024-09-15T20:37:29.166 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-326417514500 2-343597383683' 2024-09-15T20:37:29.166 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:37:29.166 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-326417514500 2024-09-15T20:37:29.166 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:37:29.167 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:37:29.168 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-326417514500 2024-09-15T20:37:29.168 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:37:29.170 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=326417514500 2024-09-15T20:37:29.170 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 326417514500 2024-09-15T20:37:29.171 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 326417514500' 2024-09-15T20:37:29.171 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:37:29.691 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 326417514498 -lt 326417514500 2024-09-15T20:37:29.691 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:37:30.692 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:37:30.693 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:37:31.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 326417514500 -lt 326417514500 2024-09-15T20:37:31.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:37:31.196 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-343597383683 2024-09-15T20:37:31.196 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:37:31.197 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:37:31.198 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-343597383683 2024-09-15T20:37:31.198 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:37:31.199 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383683 2024-09-15T20:37:31.199 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 343597383683 2024-09-15T20:37:31.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 343597383683' 2024-09-15T20:37:31.200 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:37:31.661 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383683 -lt 343597383683 2024-09-15T20:37:31.662 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:37:31.662 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:37:31.662 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:37:32.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-15T20:37:32.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:37:32.215 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:37:32.215 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:37:32.215 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:37:32.215 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:37:32.216 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:37:32.216 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:37:32.634 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:37:32.634 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:37:32.635 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:37:32.635 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:37:33.213 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-15T20:37:33.213 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T20:37:33.213 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:37:33.213 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T20:37:33.213 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:37:33.213 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:37:33.314 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:37:33.314 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:37:33.315 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:37:33.315 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:37:33.315 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:37:33.315 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:37:33.315 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:37:33.315 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:37:33.738 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:37:33.739 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:37:33.739 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:37:33.739 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:37:34.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-15T20:37:34.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:37:34.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:37:34.312 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:37:34.312 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:37:34.312 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:37:34.312 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:37:34.312 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:37:34.312 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:37:34.871 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:37:34.871 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:37:34.871 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-15T20:37:34.871 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:37:34.872 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:37:35.072 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:37:35.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:37:35.073 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:37:35.073 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:37:35.074 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:37:35.074 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:37:35.074 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:37:35.074 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:37:35.492 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:37:35.493 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:37:35.493 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:37:35.493 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:37:36.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-15T20:37:36.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:37:36.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:37:36.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:37:36.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:37:36.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:37:36.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:37:36.044 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:37:36.044 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:37:36.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:37:36.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:37:36.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-15T20:37:36.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:37:36.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T20:37:36.985 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:37:36.985 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:37:36.986 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:37:36.986 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:37:36.986 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:37:36.986 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:37:36.986 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:37:36.986 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:37:37.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:37:37.415 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:37:37.416 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:37:37.416 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:37:37.994 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-15T20:37:37.994 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:37:37.994 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:37:37.994 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:37:37.994 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:37:37.994 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:37:37.994 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:37:37.995 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:37:37.995 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:37:38.536 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:37:38.536 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:37:38.536 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-15T20:37:38.536 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:37:38.536 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-15T20:37:39.337 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:37:39.338 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:37:39.338 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:37:39.339 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:37:39.339 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:37:39.339 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:37:39.339 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:37:39.339 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:37:39.775 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:37:39.775 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:37:39.776 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:37:39.776 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:37:40.333 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-15T20:37:40.333 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:37:40.333 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:37:40.334 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:37:40.334 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:37:40.334 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:37:40.334 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:37:40.334 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:37:40.334 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:37:40.895 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:37:40.895 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:37:40.896 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-15T20:37:40.896 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:37:40.896 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-15T20:37:42.497 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:37:42.497 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:37:42.497 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:37:42.497 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:37:42.497 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:37:42.498 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:37:42.498 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:37:42.498 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:37:42.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:37:42.934 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:37:42.935 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:37:42.935 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:37:43.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-15T20:37:43.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:37:43.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:37:43.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:37:43.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:37:43.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:37:43.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:37:43.522 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:37:43.522 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:37:44.103 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:37:44.103 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:37:44.103 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-15T20:37:44.103 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:37:44.104 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-15T20:37:47.305 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:37:47.305 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:37:47.306 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:37:47.306 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:37:47.306 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:37:47.306 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:37:47.306 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:37:47.307 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:37:47.748 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:37:47.748 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:37:47.748 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:37:47.749 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:37:48.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-15T20:37:48.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:37:48.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:37:48.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:37:48.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:37:48.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:37:48.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:37:48.304 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:37:48.304 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:37:48.853 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:37:48.853 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:37:48.853 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-15T20:37:48.853 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:37:48.853 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-15T20:37:55.254 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:37:55.254 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:37:55.255 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:37:55.255 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:37:55.255 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:37:55.255 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:37:55.256 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:37:55.256 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:37:55.691 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:37:55.691 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:37:55.692 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:37:55.692 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:37:56.265 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-15T20:37:56.266 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:37:56.266 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:37:56.266 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:37:56.266 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:37:56.266 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:37:56.266 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:37:56.267 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:37:56.267 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:37:56.845 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:37:56.845 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:37:56.845 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-15T20:37:56.845 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:37:56.845 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-15T20:38:04.146 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:38:04.146 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:38:04.147 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:38:04.147 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:38:04.147 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:38:04.148 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:38:04.148 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:38:04.148 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:38:04.589 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:38:04.590 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:38:04.590 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:38:04.590 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:38:05.166 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-15T20:38:05.167 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:38:05.167 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:38:05.167 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:38:05.167 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:38:05.167 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:38:05.167 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:38:05.167 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:38:05.168 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:38:05.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:38:05.717 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:38:05.717 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-15T20:38:05.717 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-15T20:38:06.274 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:38:06.274 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_fingerprint": "f912e059-3564-4d24-9665-24ea498f0830", 2024-09-15T20:38:06.274 INFO:tasks.workunit.client.0.smithi139.stdout: "version": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:38:06.274 INFO:tasks.workunit.client.0.smithi139.stdout: "commit": "fc3e0bff7468fd55478adbae7f75508b749124b6", 2024-09-15T20:38:06.274 INFO:tasks.workunit.client.0.smithi139.stdout: "timestamp": "2024-09-15T20:38:06.276899+0000", 2024-09-15T20:38:06.275 INFO:tasks.workunit.client.0.smithi139.stdout: "tag": "", 2024-09-15T20:38:06.275 INFO:tasks.workunit.client.0.smithi139.stdout: "health": { 2024-09-15T20:38:06.275 INFO:tasks.workunit.client.0.smithi139.stdout: "status": "HEALTH_WARN", 2024-09-15T20:38:06.275 INFO:tasks.workunit.client.0.smithi139.stdout: "checks": { 2024-09-15T20:38:06.275 INFO:tasks.workunit.client.0.smithi139.stdout: "OSDMAP_FLAGS": { 2024-09-15T20:38:06.275 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:38:06.275 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:38:06.275 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-15T20:38:06.275 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 17 2024-09-15T20:38:06.275 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.275 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [], 2024-09-15T20:38:06.275 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:38:06.276 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.276 INFO:tasks.workunit.client.0.smithi139.stdout: "OSD_DOWN": { 2024-09-15T20:38:06.276 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:38:06.276 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:38:06.276 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "1 osds down", 2024-09-15T20:38:06.276 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 1 2024-09-15T20:38:06.276 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.276 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:38:06.276 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.276 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "osd.1 (root=default,host=smithi139) is down" 2024-09-15T20:38:06.276 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.276 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:06.276 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:38:06.277 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.277 INFO:tasks.workunit.client.0.smithi139.stdout: "PG_DEGRADED": { 2024-09-15T20:38:06.277 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:38:06.277 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:38:06.277 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded", 2024-09-15T20:38:06.277 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 32 2024-09-15T20:38:06.277 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.277 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:38:06.277 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.277 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-15T20:38:06.277 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.278 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.278 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [2,0]" 2024-09-15T20:38:06.278 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.278 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.278 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2024-09-15T20:38:06.278 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.278 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.278 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [2,0]" 2024-09-15T20:38:06.278 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.278 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.278 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2024-09-15T20:38:06.278 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.278 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.278 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [2,0]" 2024-09-15T20:38:06.279 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.279 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.279 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2024-09-15T20:38:06.279 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.279 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.279 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [2,0]" 2024-09-15T20:38:06.279 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.279 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.279 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [2,0]" 2024-09-15T20:38:06.279 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.279 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.279 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [2,0]" 2024-09-15T20:38:06.279 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.279 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.280 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.a is active+undersized+degraded, acting [2,0]" 2024-09-15T20:38:06.280 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.280 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.280 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2024-09-15T20:38:06.280 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.280 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.280 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2024-09-15T20:38:06.280 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.280 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.280 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.d is active+undersized+degraded, acting [2,0]" 2024-09-15T20:38:06.280 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.280 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.281 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.e is active+undersized+degraded, acting [2,0]" 2024-09-15T20:38:06.281 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.281 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.281 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.f is active+undersized+degraded, acting [2,0]" 2024-09-15T20:38:06.281 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.281 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.281 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2024-09-15T20:38:06.281 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.281 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.281 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2024-09-15T20:38:06.281 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.281 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.281 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2024-09-15T20:38:06.282 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.282 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.282 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2024-09-15T20:38:06.282 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.282 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.282 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [2,0]" 2024-09-15T20:38:06.282 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.282 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.282 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [0,2]" 2024-09-15T20:38:06.282 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.282 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.282 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2024-09-15T20:38:06.282 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.283 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.283 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [0,2]" 2024-09-15T20:38:06.283 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.283 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.283 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [2,0]" 2024-09-15T20:38:06.283 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.283 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.283 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [0,2]" 2024-09-15T20:38:06.283 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.283 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.283 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [2,0]" 2024-09-15T20:38:06.283 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.283 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.283 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2024-09-15T20:38:06.283 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.284 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.284 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2024-09-15T20:38:06.284 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.284 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.284 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [2,0]" 2024-09-15T20:38:06.284 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.284 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.284 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [2,0]" 2024-09-15T20:38:06.284 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.284 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.284 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [2,0]" 2024-09-15T20:38:06.284 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.285 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:06.285 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:38:06.285 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.285 INFO:tasks.workunit.client.0.smithi139.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-15T20:38:06.285 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:38:06.285 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:38:06.285 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-15T20:38:06.285 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 1 2024-09-15T20:38:06.285 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.285 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:38:06.285 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.285 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "application not enabled on pool 'test'" 2024-09-15T20:38:06.285 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.285 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.286 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-15T20:38:06.286 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.286 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:06.286 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:38:06.286 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.286 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.286 INFO:tasks.workunit.client.0.smithi139.stdout: "mutes": [] 2024-09-15T20:38:06.286 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.286 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap_first_committed": 1, 2024-09-15T20:38:06.286 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap_last_committed": 1, 2024-09-15T20:38:06.286 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap": { 2024-09-15T20:38:06.286 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 1, 2024-09-15T20:38:06.286 INFO:tasks.workunit.client.0.smithi139.stdout: "fsid": "0e220df1-ac9c-433e-8885-cf9f7aab293e", 2024-09-15T20:38:06.287 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:34:21.257147Z", 2024-09-15T20:38:06.287 INFO:tasks.workunit.client.0.smithi139.stdout: "created": "2024-09-15T20:34:21.257147Z", 2024-09-15T20:38:06.287 INFO:tasks.workunit.client.0.smithi139.stdout: "min_mon_release": 19, 2024-09-15T20:38:06.287 INFO:tasks.workunit.client.0.smithi139.stdout: "min_mon_release_name": "squid", 2024-09-15T20:38:06.287 INFO:tasks.workunit.client.0.smithi139.stdout: "election_strategy": 1, 2024-09-15T20:38:06.287 INFO:tasks.workunit.client.0.smithi139.stdout: "disallowed_leaders: ": "", 2024-09-15T20:38:06.287 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode": false, 2024-09-15T20:38:06.287 INFO:tasks.workunit.client.0.smithi139.stdout: "tiebreaker_mon": "", 2024-09-15T20:38:06.287 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_ranks: ": "", 2024-09-15T20:38:06.287 INFO:tasks.workunit.client.0.smithi139.stdout: "features": { 2024-09-15T20:38:06.287 INFO:tasks.workunit.client.0.smithi139.stdout: "persistent": [ 2024-09-15T20:38:06.288 INFO:tasks.workunit.client.0.smithi139.stdout: "kraken", 2024-09-15T20:38:06.288 INFO:tasks.workunit.client.0.smithi139.stdout: "luminous", 2024-09-15T20:38:06.288 INFO:tasks.workunit.client.0.smithi139.stdout: "mimic", 2024-09-15T20:38:06.288 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap-prune", 2024-09-15T20:38:06.288 INFO:tasks.workunit.client.0.smithi139.stdout: "nautilus", 2024-09-15T20:38:06.288 INFO:tasks.workunit.client.0.smithi139.stdout: "octopus", 2024-09-15T20:38:06.288 INFO:tasks.workunit.client.0.smithi139.stdout: "pacific", 2024-09-15T20:38:06.288 INFO:tasks.workunit.client.0.smithi139.stdout: "elector-pinging", 2024-09-15T20:38:06.288 INFO:tasks.workunit.client.0.smithi139.stdout: "quincy", 2024-09-15T20:38:06.288 INFO:tasks.workunit.client.0.smithi139.stdout: "reef", 2024-09-15T20:38:06.288 INFO:tasks.workunit.client.0.smithi139.stdout: "squid" 2024-09-15T20:38:06.288 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:06.289 INFO:tasks.workunit.client.0.smithi139.stdout: "optional": [] 2024-09-15T20:38:06.289 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.289 INFO:tasks.workunit.client.0.smithi139.stdout: "mons": [ 2024-09-15T20:38:06.289 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.289 INFO:tasks.workunit.client.0.smithi139.stdout: "rank": 0, 2024-09-15T20:38:06.289 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "a", 2024-09-15T20:38:06.289 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:38:06.289 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:06.289 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.289 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:06.289 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:7115", 2024-09-15T20:38:06.289 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 0 2024-09-15T20:38:06.289 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.289 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.290 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.290 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:7115/0", 2024-09-15T20:38:06.290 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-15T20:38:06.290 INFO:tasks.workunit.client.0.smithi139.stdout: "priority": 0, 2024-09-15T20:38:06.290 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 0, 2024-09-15T20:38:06.290 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_location": "{}" 2024-09-15T20:38:06.290 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.290 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.290 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.290 INFO:tasks.workunit.client.0.smithi139.stdout: "quorum": [ 2024-09-15T20:38:06.290 INFO:tasks.workunit.client.0.smithi139.stdout: 0 2024-09-15T20:38:06.290 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:06.291 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap": { 2024-09-15T20:38:06.291 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 87, 2024-09-15T20:38:06.291 INFO:tasks.workunit.client.0.smithi139.stdout: "fsid": "0e220df1-ac9c-433e-8885-cf9f7aab293e", 2024-09-15T20:38:06.291 INFO:tasks.workunit.client.0.smithi139.stdout: "created": "2024-09-15T20:34:22.232759+0000", 2024-09-15T20:38:06.291 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:37:28.945712+0000", 2024-09-15T20:38:06.291 INFO:tasks.workunit.client.0.smithi139.stdout: "last_up_change": "2024-09-15T20:37:26.014870+0000", 2024-09-15T20:38:06.291 INFO:tasks.workunit.client.0.smithi139.stdout: "last_in_change": "2024-09-15T20:37:27.456159+0000", 2024-09-15T20:38:06.291 INFO:tasks.workunit.client.0.smithi139.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T20:38:06.291 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_num": 5800384, 2024-09-15T20:38:06.291 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_set": [ 2024-09-15T20:38:06.291 INFO:tasks.workunit.client.0.smithi139.stdout: "nodown", 2024-09-15T20:38:06.291 INFO:tasks.workunit.client.0.smithi139.stdout: "noin", 2024-09-15T20:38:06.291 INFO:tasks.workunit.client.0.smithi139.stdout: "noout", 2024-09-15T20:38:06.292 INFO:tasks.workunit.client.0.smithi139.stdout: "pglog_hardlimit", 2024-09-15T20:38:06.292 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snapdirs", 2024-09-15T20:38:06.292 INFO:tasks.workunit.client.0.smithi139.stdout: "recovery_deletes", 2024-09-15T20:38:06.292 INFO:tasks.workunit.client.0.smithi139.stdout: "sortbitwise" 2024-09-15T20:38:06.292 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:06.292 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_version": 9, 2024-09-15T20:38:06.292 INFO:tasks.workunit.client.0.smithi139.stdout: "full_ratio": 0.99000000953674316, 2024-09-15T20:38:06.292 INFO:tasks.workunit.client.0.smithi139.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-15T20:38:06.292 INFO:tasks.workunit.client.0.smithi139.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T20:38:06.292 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_snapshot": "", 2024-09-15T20:38:06.292 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_max": 1, 2024-09-15T20:38:06.292 INFO:tasks.workunit.client.0.smithi139.stdout: "max_osd": 3, 2024-09-15T20:38:06.292 INFO:tasks.workunit.client.0.smithi139.stdout: "require_min_compat_client": "luminous", 2024-09-15T20:38:06.293 INFO:tasks.workunit.client.0.smithi139.stdout: "min_compat_client": "jewel", 2024-09-15T20:38:06.293 INFO:tasks.workunit.client.0.smithi139.stdout: "require_osd_release": "squid", 2024-09-15T20:38:06.293 INFO:tasks.workunit.client.0.smithi139.stdout: "allow_crimson": false, 2024-09-15T20:38:06.293 INFO:tasks.workunit.client.0.smithi139.stdout: "pools": [ 2024-09-15T20:38:06.293 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.293 INFO:tasks.workunit.client.0.smithi139.stdout: "pool": 1, 2024-09-15T20:38:06.293 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_name": "test", 2024-09-15T20:38:06.293 INFO:tasks.workunit.client.0.smithi139.stdout: "create_time": "2024-09-15T20:36:11.566515+0000", 2024-09-15T20:38:06.293 INFO:tasks.workunit.client.0.smithi139.stdout: "flags": 1, 2024-09-15T20:38:06.293 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_names": "hashpspool", 2024-09-15T20:38:06.293 INFO:tasks.workunit.client.0.smithi139.stdout: "type": 1, 2024-09-15T20:38:06.293 INFO:tasks.workunit.client.0.smithi139.stdout: "size": 3, 2024-09-15T20:38:06.293 INFO:tasks.workunit.client.0.smithi139.stdout: "min_size": 2, 2024-09-15T20:38:06.294 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_rule": 0, 2024-09-15T20:38:06.294 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_count": 0, 2024-09-15T20:38:06.294 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_target": 0, 2024-09-15T20:38:06.294 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T20:38:06.294 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T20:38:06.294 INFO:tasks.workunit.client.0.smithi139.stdout: "is_stretch_pool": false, 2024-09-15T20:38:06.294 INFO:tasks.workunit.client.0.smithi139.stdout: "object_hash": 2, 2024-09-15T20:38:06.294 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_autoscale_mode": "on", 2024-09-15T20:38:06.294 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num": 32, 2024-09-15T20:38:06.294 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_placement_num": 27, 2024-09-15T20:38:06.294 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_placement_num_target": 32, 2024-09-15T20:38:06.294 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num_target": 32, 2024-09-15T20:38:06.294 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num_pending": 32, 2024-09-15T20:38:06.295 INFO:tasks.workunit.client.0.smithi139.stdout: "last_pg_merge_meta": { 2024-09-15T20:38:06.295 INFO:tasks.workunit.client.0.smithi139.stdout: "source_pgid": "0.0", 2024-09-15T20:38:06.295 INFO:tasks.workunit.client.0.smithi139.stdout: "ready_epoch": 0, 2024-09-15T20:38:06.295 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_started": 0, 2024-09-15T20:38:06.295 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_clean": 0, 2024-09-15T20:38:06.295 INFO:tasks.workunit.client.0.smithi139.stdout: "source_version": "0'0", 2024-09-15T20:38:06.295 INFO:tasks.workunit.client.0.smithi139.stdout: "target_version": "0'0" 2024-09-15T20:38:06.295 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.295 INFO:tasks.workunit.client.0.smithi139.stdout: "last_change": "84", 2024-09-15T20:38:06.295 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend": "0", 2024-09-15T20:38:06.295 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T20:38:06.295 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend_preluminous": "33", 2024-09-15T20:38:06.295 INFO:tasks.workunit.client.0.smithi139.stdout: "auid": 0, 2024-09-15T20:38:06.295 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_mode": "selfmanaged", 2024-09-15T20:38:06.296 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_seq": 0, 2024-09-15T20:38:06.296 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_epoch": 0, 2024-09-15T20:38:06.296 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_snaps": [], 2024-09-15T20:38:06.296 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_snaps": "[]", 2024-09-15T20:38:06.296 INFO:tasks.workunit.client.0.smithi139.stdout: "quota_max_bytes": 0, 2024-09-15T20:38:06.296 INFO:tasks.workunit.client.0.smithi139.stdout: "quota_max_objects": 0, 2024-09-15T20:38:06.296 INFO:tasks.workunit.client.0.smithi139.stdout: "tiers": [], 2024-09-15T20:38:06.296 INFO:tasks.workunit.client.0.smithi139.stdout: "tier_of": -1, 2024-09-15T20:38:06.296 INFO:tasks.workunit.client.0.smithi139.stdout: "read_tier": -1, 2024-09-15T20:38:06.296 INFO:tasks.workunit.client.0.smithi139.stdout: "write_tier": -1, 2024-09-15T20:38:06.296 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_mode": "none", 2024-09-15T20:38:06.296 INFO:tasks.workunit.client.0.smithi139.stdout: "target_max_bytes": 0, 2024-09-15T20:38:06.297 INFO:tasks.workunit.client.0.smithi139.stdout: "target_max_objects": 0, 2024-09-15T20:38:06.297 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T20:38:06.297 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T20:38:06.297 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T20:38:06.297 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_min_flush_age": 0, 2024-09-15T20:38:06.297 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_min_evict_age": 0, 2024-09-15T20:38:06.297 INFO:tasks.workunit.client.0.smithi139.stdout: "erasure_code_profile": "", 2024-09-15T20:38:06.297 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_params": { 2024-09-15T20:38:06.297 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "none" 2024-09-15T20:38:06.297 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.297 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_period": 0, 2024-09-15T20:38:06.297 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_count": 0, 2024-09-15T20:38:06.297 INFO:tasks.workunit.client.0.smithi139.stdout: "use_gmt_hitset": true, 2024-09-15T20:38:06.297 INFO:tasks.workunit.client.0.smithi139.stdout: "min_read_recency_for_promote": 0, 2024-09-15T20:38:06.298 INFO:tasks.workunit.client.0.smithi139.stdout: "min_write_recency_for_promote": 0, 2024-09-15T20:38:06.298 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T20:38:06.298 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_search_last_n": 0, 2024-09-15T20:38:06.298 INFO:tasks.workunit.client.0.smithi139.stdout: "grade_table": [], 2024-09-15T20:38:06.298 INFO:tasks.workunit.client.0.smithi139.stdout: "stripe_width": 0, 2024-09-15T20:38:06.298 INFO:tasks.workunit.client.0.smithi139.stdout: "expected_num_objects": 0, 2024-09-15T20:38:06.298 INFO:tasks.workunit.client.0.smithi139.stdout: "fast_read": false, 2024-09-15T20:38:06.298 INFO:tasks.workunit.client.0.smithi139.stdout: "options": {}, 2024-09-15T20:38:06.298 INFO:tasks.workunit.client.0.smithi139.stdout: "application_metadata": {}, 2024-09-15T20:38:06.298 INFO:tasks.workunit.client.0.smithi139.stdout: "read_balance": { 2024-09-15T20:38:06.298 INFO:tasks.workunit.client.0.smithi139.stdout: "score_type": "Fair distribution", 2024-09-15T20:38:06.298 INFO:tasks.workunit.client.0.smithi139.stdout: "score_acting": 1.4900000095367432, 2024-09-15T20:38:06.298 INFO:tasks.workunit.client.0.smithi139.stdout: "score_stable": 1.4900000095367432, 2024-09-15T20:38:06.298 INFO:tasks.workunit.client.0.smithi139.stdout: "optimal_score": 0.67000001668930054, 2024-09-15T20:38:06.298 INFO:tasks.workunit.client.0.smithi139.stdout: "raw_score_acting": 1, 2024-09-15T20:38:06.299 INFO:tasks.workunit.client.0.smithi139.stdout: "raw_score_stable": 1, 2024-09-15T20:38:06.299 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity_weighted": 1, 2024-09-15T20:38:06.299 INFO:tasks.workunit.client.0.smithi139.stdout: "average_primary_affinity": 1, 2024-09-15T20:38:06.299 INFO:tasks.workunit.client.0.smithi139.stdout: "average_primary_affinity_weighted": 1 2024-09-15T20:38:06.299 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.299 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.299 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:06.299 INFO:tasks.workunit.client.0.smithi139.stdout: "osds": [ 2024-09-15T20:38:06.299 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.299 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:38:06.299 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "8ceaad64-e87a-460e-8e98-3caf0f7eff12", 2024-09-15T20:38:06.299 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 1, 2024-09-15T20:38:06.299 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:38:06.299 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:38:06.299 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:38:06.300 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 5, 2024-09-15T20:38:06.300 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 54, 2024-09-15T20:38:06.300 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 76, 2024-09-15T20:38:06.300 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 83, 2024-09-15T20:38:06.300 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 66, 2024-09-15T20:38:06.300 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:38:06.300 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:38:06.300 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:06.300 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.300 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:06.300 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6802", 2024-09-15T20:38:06.300 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3684664553 2024-09-15T20:38:06.300 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.301 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.301 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:06.301 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6803", 2024-09-15T20:38:06.301 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3684664553 2024-09-15T20:38:06.301 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.301 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.301 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.301 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:38:06.301 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:06.301 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.301 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:06.301 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6804", 2024-09-15T20:38:06.301 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3684664553 2024-09-15T20:38:06.301 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.302 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.302 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:06.302 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6805", 2024-09-15T20:38:06.302 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3684664553 2024-09-15T20:38:06.302 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.302 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.302 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.302 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:38:06.302 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:06.302 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.302 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:06.302 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6808", 2024-09-15T20:38:06.302 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3684664553 2024-09-15T20:38:06.302 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.303 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.303 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:06.303 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6809", 2024-09-15T20:38:06.303 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3684664553 2024-09-15T20:38:06.303 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.303 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.303 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.303 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:38:06.303 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:06.303 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.303 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:06.303 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6806", 2024-09-15T20:38:06.303 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3684664553 2024-09-15T20:38:06.303 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.304 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.304 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:06.304 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6807", 2024-09-15T20:38:06.304 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3684664553 2024-09-15T20:38:06.304 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.304 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.304 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.304 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6803/3684664553", 2024-09-15T20:38:06.304 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6805/3684664553", 2024-09-15T20:38:06.304 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3684664553", 2024-09-15T20:38:06.304 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3684664553", 2024-09-15T20:38:06.304 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:38:06.305 INFO:tasks.workunit.client.0.smithi139.stdout: "exists", 2024-09-15T20:38:06.305 INFO:tasks.workunit.client.0.smithi139.stdout: "up" 2024-09-15T20:38:06.305 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.305 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.305 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.305 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:38:06.305 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "d5b37d03-d9a9-4f3f-8c3e-d61cdbdce38e", 2024-09-15T20:38:06.305 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 0, 2024-09-15T20:38:06.305 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:38:06.305 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:38:06.305 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:38:06.305 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 0, 2024-09-15T20:38:06.305 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 0, 2024-09-15T20:38:06.306 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 10, 2024-09-15T20:38:06.306 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 57, 2024-09-15T20:38:06.306 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 67, 2024-09-15T20:38:06.306 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:38:06.306 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:38:06.306 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:06.306 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.306 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:06.306 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6810", 2024-09-15T20:38:06.306 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 351107530 2024-09-15T20:38:06.306 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.306 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.306 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:06.306 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6811", 2024-09-15T20:38:06.307 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 351107530 2024-09-15T20:38:06.307 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.307 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.307 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.307 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:38:06.307 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:06.307 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.307 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:06.307 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6812", 2024-09-15T20:38:06.307 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 351107530 2024-09-15T20:38:06.308 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.308 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.308 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:06.308 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6813", 2024-09-15T20:38:06.308 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 351107530 2024-09-15T20:38:06.308 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.308 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.308 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.308 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:38:06.308 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:06.308 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.308 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:06.309 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6816", 2024-09-15T20:38:06.309 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 351107530 2024-09-15T20:38:06.309 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.309 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.309 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:06.309 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6817", 2024-09-15T20:38:06.309 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 351107530 2024-09-15T20:38:06.309 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.309 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.309 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.309 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:38:06.310 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:06.310 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.310 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:06.310 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6814", 2024-09-15T20:38:06.310 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 351107530 2024-09-15T20:38:06.310 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.310 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.310 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:06.310 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6815", 2024-09-15T20:38:06.310 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 351107530 2024-09-15T20:38:06.310 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.310 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.311 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.311 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6811/351107530", 2024-09-15T20:38:06.311 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6813/351107530", 2024-09-15T20:38:06.311 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6817/351107530", 2024-09-15T20:38:06.311 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6815/351107530", 2024-09-15T20:38:06.311 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:38:06.311 INFO:tasks.workunit.client.0.smithi139.stdout: "exists" 2024-09-15T20:38:06.311 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.311 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.311 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.312 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:38:06.312 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "5347fe2a-12cd-44fe-9cd9-0dbc2bc7ebb8", 2024-09-15T20:38:06.312 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 1, 2024-09-15T20:38:06.312 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:38:06.312 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:38:06.312 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:38:06.312 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 15, 2024-09-15T20:38:06.312 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 53, 2024-09-15T20:38:06.312 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 80, 2024-09-15T20:38:06.312 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 85, 2024-09-15T20:38:06.312 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 68, 2024-09-15T20:38:06.312 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:38:06.313 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:38:06.313 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:06.313 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.313 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:06.313 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6810", 2024-09-15T20:38:06.313 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3676612316 2024-09-15T20:38:06.313 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.313 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.313 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:06.313 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6811", 2024-09-15T20:38:06.313 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3676612316 2024-09-15T20:38:06.314 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.314 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.314 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.314 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:38:06.314 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:06.314 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.314 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:06.314 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6812", 2024-09-15T20:38:06.314 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3676612316 2024-09-15T20:38:06.314 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.315 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.315 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:06.315 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6813", 2024-09-15T20:38:06.315 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3676612316 2024-09-15T20:38:06.315 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.315 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.315 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.315 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:38:06.315 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:06.316 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.316 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:06.316 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6816", 2024-09-15T20:38:06.316 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3676612316 2024-09-15T20:38:06.316 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.316 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.316 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:06.316 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6817", 2024-09-15T20:38:06.316 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3676612316 2024-09-15T20:38:06.316 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.317 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.317 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.317 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:38:06.317 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:06.317 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.317 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:06.317 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6814", 2024-09-15T20:38:06.317 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3676612316 2024-09-15T20:38:06.317 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.317 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.318 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:06.318 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6815", 2024-09-15T20:38:06.318 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3676612316 2024-09-15T20:38:06.318 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.318 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.318 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.318 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6811/3676612316", 2024-09-15T20:38:06.318 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6813/3676612316", 2024-09-15T20:38:06.318 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3676612316", 2024-09-15T20:38:06.318 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3676612316", 2024-09-15T20:38:06.318 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:38:06.318 INFO:tasks.workunit.client.0.smithi139.stdout: "exists", 2024-09-15T20:38:06.318 INFO:tasks.workunit.client.0.smithi139.stdout: "up" 2024-09-15T20:38:06.318 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.319 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.319 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:06.319 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_xinfo": [ 2024-09-15T20:38:06.319 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.319 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:38:06.319 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:37:12.291089+0000", 2024-09-15T20:38:06.319 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:38:06.319 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:38:06.319 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:38:06.319 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:38:06.319 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:34:55.253095+0000", 2024-09-15T20:38:06.319 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:38:06.319 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.319 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.319 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:38:06.320 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:37:12.865088+0000", 2024-09-15T20:38:06.320 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:38:06.320 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:38:06.320 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:38:06.320 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:38:06.320 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:35:31.139386+0000", 2024-09-15T20:38:06.320 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:38:06.320 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.320 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.320 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:38:06.320 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:37:13.435960+0000", 2024-09-15T20:38:06.320 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:38:06.320 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:38:06.321 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:38:06.321 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:38:06.321 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:36:07.391280+0000", 2024-09-15T20:38:06.321 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:38:06.321 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.321 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:06.321 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap": [], 2024-09-15T20:38:06.321 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap_items": [], 2024-09-15T20:38:06.321 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap_primaries": [], 2024-09-15T20:38:06.321 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_temp": [], 2024-09-15T20:38:06.321 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_temp": [], 2024-09-15T20:38:06.321 INFO:tasks.workunit.client.0.smithi139.stdout: "blocklist": {}, 2024-09-15T20:38:06.321 INFO:tasks.workunit.client.0.smithi139.stdout: "range_blocklist": {}, 2024-09-15T20:38:06.321 INFO:tasks.workunit.client.0.smithi139.stdout: "erasure_code_profiles": { 2024-09-15T20:38:06.322 INFO:tasks.workunit.client.0.smithi139.stdout: "default": { 2024-09-15T20:38:06.322 INFO:tasks.workunit.client.0.smithi139.stdout: "k": "2", 2024-09-15T20:38:06.322 INFO:tasks.workunit.client.0.smithi139.stdout: "m": "2", 2024-09-15T20:38:06.322 INFO:tasks.workunit.client.0.smithi139.stdout: "plugin": "jerasure", 2024-09-15T20:38:06.322 INFO:tasks.workunit.client.0.smithi139.stdout: "technique": "reed_sol_van" 2024-09-15T20:38:06.322 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.322 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.322 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_snaps_queue": [], 2024-09-15T20:38:06.322 INFO:tasks.workunit.client.0.smithi139.stdout: "new_removed_snaps": [], 2024-09-15T20:38:06.322 INFO:tasks.workunit.client.0.smithi139.stdout: "new_purged_snaps": [], 2024-09-15T20:38:06.322 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_node_flags": {}, 2024-09-15T20:38:06.322 INFO:tasks.workunit.client.0.smithi139.stdout: "device_class_flags": {}, 2024-09-15T20:38:06.322 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode": { 2024-09-15T20:38:06.323 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode_enabled": false, 2024-09-15T20:38:06.323 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_bucket_count": 0, 2024-09-15T20:38:06.323 INFO:tasks.workunit.client.0.smithi139.stdout: "degraded_stretch_mode": 0, 2024-09-15T20:38:06.323 INFO:tasks.workunit.client.0.smithi139.stdout: "recovering_stretch_mode": 0, 2024-09-15T20:38:06.323 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode_bucket": 0 2024-09-15T20:38:06.323 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.323 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.323 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_metadata": [ 2024-09-15T20:38:06.323 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.323 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:38:06.323 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:38:06.323 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6804/3684664553,v1:127.0.0.1:6805/3684664553]", 2024-09-15T20:38:06.324 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:38:06.324 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:38:06.324 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:38:06.324 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:38:06.324 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:38:06.324 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:38:06.324 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:38:06.324 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:38:06.324 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:38:06.324 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:38:06.324 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:38:06.325 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-15T20:38:06.325 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:38:06.325 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:38:06.325 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:38:06.325 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:38:06.325 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:38:06.325 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:38:06.325 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:38:06.325 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:38:06.325 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:38:06.325 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:38:06.325 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:34:28.822135Z", 2024-09-15T20:38:06.325 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:38:06.326 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:38:06.326 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:38:06.326 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:38:06.326 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:38:06.326 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:38:06.326 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:38:06.326 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6802/3684664553,v1:127.0.0.1:6803/3684664553]", 2024-09-15T20:38:06.326 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:38:06.326 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3684664553,v1:127.0.0.1:6809/3684664553]", 2024-09-15T20:38:06.326 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3684664553,v1:127.0.0.1:6807/3684664553]", 2024-09-15T20:38:06.326 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:38:06.326 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:38:06.327 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:38:06.327 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:38:06.327 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:38:06.327 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:38:06.327 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:38:06.327 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:38:06.327 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:38:06.327 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/0", 2024-09-15T20:38:06.327 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:38:06.327 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:38:06.327 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:38:06.327 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.327 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.328 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:38:06.328 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:38:06.328 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6812/351107530,v1:127.0.0.1:6813/351107530]", 2024-09-15T20:38:06.328 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:38:06.328 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:38:06.328 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:38:06.328 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:38:06.328 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:38:06.328 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:38:06.328 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:38:06.328 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:38:06.328 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:38:06.328 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:38:06.329 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:38:06.329 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-15T20:38:06.329 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:38:06.329 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:38:06.329 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:38:06.329 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:38:06.329 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:38:06.329 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:38:06.329 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:38:06.329 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:38:06.329 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:38:06.330 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:38:06.330 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:35:03.011496Z", 2024-09-15T20:38:06.330 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:38:06.330 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:38:06.330 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:38:06.330 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:38:06.330 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:38:06.330 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:38:06.330 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:38:06.330 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6810/351107530,v1:127.0.0.1:6811/351107530]", 2024-09-15T20:38:06.330 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:38:06.330 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/351107530,v1:127.0.0.1:6817/351107530]", 2024-09-15T20:38:06.330 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/351107530,v1:127.0.0.1:6815/351107530]", 2024-09-15T20:38:06.331 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:38:06.331 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:38:06.331 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:38:06.331 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:38:06.331 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:38:06.331 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:38:06.331 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:38:06.331 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:38:06.331 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:38:06.331 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/1", 2024-09-15T20:38:06.331 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:38:06.331 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:38:06.331 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:38:06.331 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.332 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.332 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:38:06.332 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:38:06.332 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6812/3676612316,v1:127.0.0.1:6813/3676612316]", 2024-09-15T20:38:06.332 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:38:06.332 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:38:06.332 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:38:06.332 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:38:06.332 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:38:06.332 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:38:06.332 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:38:06.332 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:38:06.333 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:38:06.333 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:38:06.333 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:38:06.333 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-15T20:38:06.333 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:38:06.333 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:38:06.333 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:38:06.333 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:38:06.333 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:38:06.333 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:38:06.333 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:38:06.333 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:38:06.333 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:38:06.333 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:38:06.334 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:35:38.971291Z", 2024-09-15T20:38:06.334 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:38:06.334 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:38:06.334 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:38:06.334 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:38:06.334 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:38:06.334 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:38:06.334 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:38:06.334 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6810/3676612316,v1:127.0.0.1:6811/3676612316]", 2024-09-15T20:38:06.334 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:38:06.334 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3676612316,v1:127.0.0.1:6817/3676612316]", 2024-09-15T20:38:06.334 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3676612316,v1:127.0.0.1:6815/3676612316]", 2024-09-15T20:38:06.334 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:38:06.335 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:38:06.335 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:38:06.335 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:38:06.335 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:38:06.335 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:38:06.335 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:38:06.335 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:38:06.335 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:38:06.335 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/2", 2024-09-15T20:38:06.335 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:38:06.335 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:38:06.335 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:38:06.335 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.335 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:06.335 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_clean_epochs": { 2024-09-15T20:38:06.336 INFO:tasks.workunit.client.0.smithi139.stdout: "min_last_epoch_clean": 0, 2024-09-15T20:38:06.336 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_clean": { 2024-09-15T20:38:06.336 INFO:tasks.workunit.client.0.smithi139.stdout: "per_pool": [ 2024-09-15T20:38:06.336 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.336 INFO:tasks.workunit.client.0.smithi139.stdout: "poolid": 1, 2024-09-15T20:38:06.336 INFO:tasks.workunit.client.0.smithi139.stdout: "floor": 36 2024-09-15T20:38:06.336 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.336 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.336 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.336 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_epochs": [ 2024-09-15T20:38:06.337 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.337 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:38:06.337 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 76 2024-09-15T20:38:06.337 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.337 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.337 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:38:06.337 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 80 2024-09-15T20:38:06.337 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.337 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.337 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.337 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_first_committed": 1, 2024-09-15T20:38:06.337 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_last_committed": 87, 2024-09-15T20:38:06.338 INFO:tasks.workunit.client.0.smithi139.stdout: "crushmap": { 2024-09-15T20:38:06.338 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": [ 2024-09-15T20:38:06.338 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.338 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:38:06.338 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.0", 2024-09-15T20:38:06.338 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:38:06.338 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.338 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.338 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:38:06.338 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.1", 2024-09-15T20:38:06.338 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:38:06.338 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.339 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.339 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:38:06.339 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.2", 2024-09-15T20:38:06.339 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:38:06.339 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.339 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:06.339 INFO:tasks.workunit.client.0.smithi139.stdout: "types": [ 2024-09-15T20:38:06.339 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.339 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 0, 2024-09-15T20:38:06.340 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd" 2024-09-15T20:38:06.340 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.340 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.340 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:38:06.340 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "host" 2024-09-15T20:38:06.340 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.340 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.340 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 2, 2024-09-15T20:38:06.340 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "chassis" 2024-09-15T20:38:06.340 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.340 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.340 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 3, 2024-09-15T20:38:06.341 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "rack" 2024-09-15T20:38:06.341 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.341 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.341 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 4, 2024-09-15T20:38:06.341 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "row" 2024-09-15T20:38:06.341 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.341 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.341 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 5, 2024-09-15T20:38:06.341 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "pdu" 2024-09-15T20:38:06.341 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.341 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.341 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 6, 2024-09-15T20:38:06.342 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "pod" 2024-09-15T20:38:06.342 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.342 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.342 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 7, 2024-09-15T20:38:06.342 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "room" 2024-09-15T20:38:06.342 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.342 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.342 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 8, 2024-09-15T20:38:06.342 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "datacenter" 2024-09-15T20:38:06.342 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.342 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.342 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 9, 2024-09-15T20:38:06.342 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "zone" 2024-09-15T20:38:06.343 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.343 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.343 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 10, 2024-09-15T20:38:06.343 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "region" 2024-09-15T20:38:06.343 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.343 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.343 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:38:06.343 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "root" 2024-09-15T20:38:06.343 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.343 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:06.343 INFO:tasks.workunit.client.0.smithi139.stdout: "buckets": [ 2024-09-15T20:38:06.343 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.343 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -1, 2024-09-15T20:38:06.344 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "default", 2024-09-15T20:38:06.344 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:38:06.344 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "root", 2024-09-15T20:38:06.344 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:38:06.344 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:38:06.344 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:38:06.344 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:38:06.344 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.344 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -3, 2024-09-15T20:38:06.344 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:38:06.344 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:38:06.344 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.344 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.345 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.345 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.345 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -2, 2024-09-15T20:38:06.345 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "default~hdd", 2024-09-15T20:38:06.345 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:38:06.345 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "root", 2024-09-15T20:38:06.345 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:38:06.345 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:38:06.345 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:38:06.345 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:38:06.345 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.345 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -4, 2024-09-15T20:38:06.346 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:38:06.346 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:38:06.346 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.346 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.346 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.346 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.346 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -3, 2024-09-15T20:38:06.346 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "smithi139", 2024-09-15T20:38:06.346 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:38:06.346 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "host", 2024-09-15T20:38:06.346 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:38:06.346 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:38:06.346 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:38:06.347 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:38:06.347 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.347 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:38:06.347 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:38:06.347 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:38:06.347 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.347 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.347 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:38:06.347 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:38:06.347 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 1 2024-09-15T20:38:06.347 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.347 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.347 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:38:06.348 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:38:06.348 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 2 2024-09-15T20:38:06.348 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.348 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.348 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.348 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.348 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -4, 2024-09-15T20:38:06.348 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "smithi139~hdd", 2024-09-15T20:38:06.348 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:38:06.348 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "host", 2024-09-15T20:38:06.348 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:38:06.348 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:38:06.348 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:38:06.349 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:38:06.349 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.349 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:38:06.349 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:38:06.349 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:38:06.349 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.349 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.349 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:38:06.349 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:38:06.349 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 1 2024-09-15T20:38:06.349 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.349 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.349 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:38:06.349 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:38:06.350 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 2 2024-09-15T20:38:06.350 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.350 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.350 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.350 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:06.350 INFO:tasks.workunit.client.0.smithi139.stdout: "rules": [ 2024-09-15T20:38:06.350 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.350 INFO:tasks.workunit.client.0.smithi139.stdout: "rule_id": 0, 2024-09-15T20:38:06.350 INFO:tasks.workunit.client.0.smithi139.stdout: "rule_name": "replicated_rule", 2024-09-15T20:38:06.350 INFO:tasks.workunit.client.0.smithi139.stdout: "type": 1, 2024-09-15T20:38:06.350 INFO:tasks.workunit.client.0.smithi139.stdout: "steps": [ 2024-09-15T20:38:06.350 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.350 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "take", 2024-09-15T20:38:06.350 INFO:tasks.workunit.client.0.smithi139.stdout: "item": -1, 2024-09-15T20:38:06.351 INFO:tasks.workunit.client.0.smithi139.stdout: "item_name": "default" 2024-09-15T20:38:06.351 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.351 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.351 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "choose_firstn", 2024-09-15T20:38:06.351 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 0, 2024-09-15T20:38:06.351 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "osd" 2024-09-15T20:38:06.351 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.351 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.351 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "emit" 2024-09-15T20:38:06.351 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.351 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:06.351 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.351 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:06.351 INFO:tasks.workunit.client.0.smithi139.stdout: "tunables": { 2024-09-15T20:38:06.352 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_local_tries": 0, 2024-09-15T20:38:06.352 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_local_fallback_tries": 0, 2024-09-15T20:38:06.352 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_total_tries": 50, 2024-09-15T20:38:06.352 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_descend_once": 1, 2024-09-15T20:38:06.352 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_vary_r": 1, 2024-09-15T20:38:06.352 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_stable": 1, 2024-09-15T20:38:06.352 INFO:tasks.workunit.client.0.smithi139.stdout: "msr_descents": 100, 2024-09-15T20:38:06.352 INFO:tasks.workunit.client.0.smithi139.stdout: "msr_collision_tries": 100, 2024-09-15T20:38:06.353 INFO:tasks.workunit.client.0.smithi139.stdout: "straw_calc_version": 1, 2024-09-15T20:38:06.353 INFO:tasks.workunit.client.0.smithi139.stdout: "allowed_bucket_algs": 54, 2024-09-15T20:38:06.353 INFO:tasks.workunit.client.0.smithi139.stdout: "profile": "jewel", 2024-09-15T20:38:06.353 INFO:tasks.workunit.client.0.smithi139.stdout: "optimal_tunables": 1, 2024-09-15T20:38:06.353 INFO:tasks.workunit.client.0.smithi139.stdout: "legacy_tunables": 0, 2024-09-15T20:38:06.353 INFO:tasks.workunit.client.0.smithi139.stdout: "minimum_required_version": "jewel", 2024-09-15T20:38:06.353 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables": 1, 2024-09-15T20:38:06.353 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables2": 1, 2024-09-15T20:38:06.353 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v2_rules": 0, 2024-09-15T20:38:06.354 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables3": 1, 2024-09-15T20:38:06.354 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v3_rules": 0, 2024-09-15T20:38:06.354 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v4_buckets": 1, 2024-09-15T20:38:06.354 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables5": 1, 2024-09-15T20:38:06.354 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v5_rules": 0, 2024-09-15T20:38:06.354 INFO:tasks.workunit.client.0.smithi139.stdout: "has_msr_rules": 0 2024-09-15T20:38:06.354 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.354 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_args": {} 2024-09-15T20:38:06.354 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.354 INFO:tasks.workunit.client.0.smithi139.stdout: "fsmap": { 2024-09-15T20:38:06.354 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 1, 2024-09-15T20:38:06.354 INFO:tasks.workunit.client.0.smithi139.stdout: "btime": "2024-09-15T20:34:22:211262+0000", 2024-09-15T20:38:06.355 INFO:tasks.workunit.client.0.smithi139.stdout: "default_fscid": -1, 2024-09-15T20:38:06.355 INFO:tasks.workunit.client.0.smithi139.stdout: "compat": { 2024-09-15T20:38:06.355 INFO:tasks.workunit.client.0.smithi139.stdout: "compat": {}, 2024-09-15T20:38:06.355 INFO:tasks.workunit.client.0.smithi139.stdout: "ro_compat": {}, 2024-09-15T20:38:06.355 INFO:tasks.workunit.client.0.smithi139.stdout: "incompat": { 2024-09-15T20:38:06.355 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_1": "base v0.20", 2024-09-15T20:38:06.355 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_2": "client writeable ranges", 2024-09-15T20:38:06.355 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_3": "default file layouts on dirs", 2024-09-15T20:38:06.355 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_4": "dir inode in separate object", 2024-09-15T20:38:06.355 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_5": "mds uses versioned encoding", 2024-09-15T20:38:06.355 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-15T20:38:06.355 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_8": "no anchor table", 2024-09-15T20:38:06.356 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_9": "file layout v2", 2024-09-15T20:38:06.356 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_10": "snaprealm v2", 2024-09-15T20:38:06.356 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_11": "minor log segments", 2024-09-15T20:38:06.356 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_12": "quiesce subvolumes" 2024-09-15T20:38:06.356 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.356 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.356 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_flags": { 2024-09-15T20:38:06.356 INFO:tasks.workunit.client.0.smithi139.stdout: "enable_multiple": true, 2024-09-15T20:38:06.356 INFO:tasks.workunit.client.0.smithi139.stdout: "ever_enabled_multiple": true 2024-09-15T20:38:06.356 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.356 INFO:tasks.workunit.client.0.smithi139.stdout: "standbys": [], 2024-09-15T20:38:06.356 INFO:tasks.workunit.client.0.smithi139.stdout: "filesystems": [] 2024-09-15T20:38:06.356 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.357 INFO:tasks.workunit.client.0.smithi139.stdout: "mdsmap_first_committed": 1, 2024-09-15T20:38:06.357 INFO:tasks.workunit.client.0.smithi139.stdout: "mdsmap_last_committed": 1, 2024-09-15T20:38:06.357 INFO:tasks.workunit.client.0.smithi139.stdout: "auth": { 2024-09-15T20:38:06.357 INFO:tasks.workunit.client.0.smithi139.stdout: "first_committed": 1, 2024-09-15T20:38:06.357 INFO:tasks.workunit.client.0.smithi139.stdout: "last_committed": 5, 2024-09-15T20:38:06.357 INFO:tasks.workunit.client.0.smithi139.stdout: "num_secrets": 3 2024-09-15T20:38:06.357 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.357 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg": 32, 2024-09-15T20:38:06.357 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_active": 32, 2024-09-15T20:38:06.357 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_unknown": 0, 2024-09-15T20:38:06.357 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osd": 3, 2024-09-15T20:38:06.357 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_sum": { 2024-09-15T20:38:06.358 INFO:tasks.workunit.client.0.smithi139.stdout: "stat_sum": { 2024-09-15T20:38:06.358 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes": 60833, 2024-09-15T20:38:06.358 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects": 100, 2024-09-15T20:38:06.358 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_clones": 0, 2024-09-15T20:38:06.358 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_copies": 300, 2024-09-15T20:38:06.358 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T20:38:06.358 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing": 0, 2024-09-15T20:38:06.358 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_degraded": 100, 2024-09-15T20:38:06.358 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_misplaced": 0, 2024-09-15T20:38:06.358 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_unfound": 0, 2024-09-15T20:38:06.358 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_dirty": 100, 2024-09-15T20:38:06.358 INFO:tasks.workunit.client.0.smithi139.stdout: "num_whiteouts": 0, 2024-09-15T20:38:06.358 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read": 0, 2024-09-15T20:38:06.359 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read_kb": 0, 2024-09-15T20:38:06.359 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write": 58, 2024-09-15T20:38:06.359 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write_kb": 58, 2024-09-15T20:38:06.359 INFO:tasks.workunit.client.0.smithi139.stdout: "num_scrub_errors": 0, 2024-09-15T20:38:06.359 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T20:38:06.359 INFO:tasks.workunit.client.0.smithi139.stdout: "num_deep_scrub_errors": 0, 2024-09-15T20:38:06.359 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_recovered": 0, 2024-09-15T20:38:06.359 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_recovered": 0, 2024-09-15T20:38:06.359 INFO:tasks.workunit.client.0.smithi139.stdout: "num_keys_recovered": 0, 2024-09-15T20:38:06.359 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_omap": 0, 2024-09-15T20:38:06.359 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T20:38:06.359 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T20:38:06.360 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush": 0, 2024-09-15T20:38:06.360 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_kb": 0, 2024-09-15T20:38:06.360 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict": 0, 2024-09-15T20:38:06.360 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_kb": 0, 2024-09-15T20:38:06.360 INFO:tasks.workunit.client.0.smithi139.stdout: "num_promote": 0, 2024-09-15T20:38:06.360 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_high": 0, 2024-09-15T20:38:06.360 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_low": 0, 2024-09-15T20:38:06.360 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_some": 0, 2024-09-15T20:38:06.360 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_full": 0, 2024-09-15T20:38:06.360 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_pinned": 0, 2024-09-15T20:38:06.360 INFO:tasks.workunit.client.0.smithi139.stdout: "num_legacy_snapsets": 0, 2024-09-15T20:38:06.360 INFO:tasks.workunit.client.0.smithi139.stdout: "num_large_omap_objects": 0, 2024-09-15T20:38:06.360 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_manifest": 0, 2024-09-15T20:38:06.361 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_bytes": 0, 2024-09-15T20:38:06.361 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_keys": 0, 2024-09-15T20:38:06.361 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_repaired": 0 2024-09-15T20:38:06.361 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.361 INFO:tasks.workunit.client.0.smithi139.stdout: "store_stats": { 2024-09-15T20:38:06.361 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 0, 2024-09-15T20:38:06.361 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 0, 2024-09-15T20:38:06.361 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:38:06.361 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 0, 2024-09-15T20:38:06.361 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 0, 2024-09-15T20:38:06.361 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:38:06.361 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:38:06.362 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:38:06.362 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 0, 2024-09-15T20:38:06.362 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 0 2024-09-15T20:38:06.362 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.362 INFO:tasks.workunit.client.0.smithi139.stdout: "log_size": 553, 2024-09-15T20:38:06.362 INFO:tasks.workunit.client.0.smithi139.stdout: "ondisk_log_size": 553, 2024-09-15T20:38:06.362 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 64, 2024-09-15T20:38:06.362 INFO:tasks.workunit.client.0.smithi139.stdout: "acting": 64, 2024-09-15T20:38:06.362 INFO:tasks.workunit.client.0.smithi139.stdout: "num_store_stats": 0 2024-09-15T20:38:06.362 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.362 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_sum": { 2024-09-15T20:38:06.362 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 0, 2024-09-15T20:38:06.362 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 0, 2024-09-15T20:38:06.362 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pgs": 96, 2024-09-15T20:38:06.363 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osds": 3, 2024-09-15T20:38:06.363 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_osds": 3, 2024-09-15T20:38:06.363 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T20:38:06.363 INFO:tasks.workunit.client.0.smithi139.stdout: "kb": 314572800, 2024-09-15T20:38:06.363 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used": 84640, 2024-09-15T20:38:06.363 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_data": 3088, 2024-09-15T20:38:06.363 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_omap": 10, 2024-09-15T20:38:06.363 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_meta": 81461, 2024-09-15T20:38:06.363 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_avail": 314488160, 2024-09-15T20:38:06.363 INFO:tasks.workunit.client.0.smithi139.stdout: "statfs": { 2024-09-15T20:38:06.363 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 322122547200, 2024-09-15T20:38:06.363 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 322035875840, 2024-09-15T20:38:06.364 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:38:06.364 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 3162112, 2024-09-15T20:38:06.364 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 823654, 2024-09-15T20:38:06.364 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:38:06.364 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:38:06.364 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:38:06.364 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 10399, 2024-09-15T20:38:06.364 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 83416929 2024-09-15T20:38:06.364 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.364 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_peers": [], 2024-09-15T20:38:06.364 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_trim_queue_len": 0, 2024-09-15T20:38:06.364 INFO:tasks.workunit.client.0.smithi139.stdout: "num_snap_trimming": 0, 2024-09-15T20:38:06.365 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shards_repaired": 0, 2024-09-15T20:38:06.365 INFO:tasks.workunit.client.0.smithi139.stdout: "op_queue_age_hist": { 2024-09-15T20:38:06.365 INFO:tasks.workunit.client.0.smithi139.stdout: "histogram": [], 2024-09-15T20:38:06.365 INFO:tasks.workunit.client.0.smithi139.stdout: "upper_bound": 1 2024-09-15T20:38:06.365 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.365 INFO:tasks.workunit.client.0.smithi139.stdout: "perf_stat": { 2024-09-15T20:38:06.365 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ms": 100, 2024-09-15T20:38:06.365 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ms": 100, 2024-09-15T20:38:06.365 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ns": 100000000, 2024-09-15T20:38:06.365 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ns": 100000000 2024-09-15T20:38:06.365 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.365 INFO:tasks.workunit.client.0.smithi139.stdout: "alerts": [], 2024-09-15T20:38:06.365 INFO:tasks.workunit.client.0.smithi139.stdout: "network_ping_times": [] 2024-09-15T20:38:06.366 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.366 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_sum_by_class": { 2024-09-15T20:38:06.366 INFO:tasks.workunit.client.0.smithi139.stdout: "hdd": { 2024-09-15T20:38:06.366 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 0, 2024-09-15T20:38:06.366 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 0, 2024-09-15T20:38:06.366 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pgs": 96, 2024-09-15T20:38:06.366 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osds": 3, 2024-09-15T20:38:06.366 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_osds": 3, 2024-09-15T20:38:06.366 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T20:38:06.366 INFO:tasks.workunit.client.0.smithi139.stdout: "kb": 314572800, 2024-09-15T20:38:06.366 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used": 84640, 2024-09-15T20:38:06.366 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_data": 3088, 2024-09-15T20:38:06.366 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_omap": 10, 2024-09-15T20:38:06.367 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_meta": 81461, 2024-09-15T20:38:06.367 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_avail": 314488160, 2024-09-15T20:38:06.367 INFO:tasks.workunit.client.0.smithi139.stdout: "statfs": { 2024-09-15T20:38:06.367 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 322122547200, 2024-09-15T20:38:06.367 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 322035875840, 2024-09-15T20:38:06.367 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:38:06.367 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 3162112, 2024-09-15T20:38:06.367 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 823654, 2024-09-15T20:38:06.367 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:38:06.367 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:38:06.367 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:38:06.367 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 10399, 2024-09-15T20:38:06.367 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 83416929 2024-09-15T20:38:06.368 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.368 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_peers": [], 2024-09-15T20:38:06.368 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_trim_queue_len": 0, 2024-09-15T20:38:06.368 INFO:tasks.workunit.client.0.smithi139.stdout: "num_snap_trimming": 0, 2024-09-15T20:38:06.368 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shards_repaired": 0, 2024-09-15T20:38:06.368 INFO:tasks.workunit.client.0.smithi139.stdout: "op_queue_age_hist": { 2024-09-15T20:38:06.368 INFO:tasks.workunit.client.0.smithi139.stdout: "histogram": [], 2024-09-15T20:38:06.368 INFO:tasks.workunit.client.0.smithi139.stdout: "upper_bound": 1 2024-09-15T20:38:06.368 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.368 INFO:tasks.workunit.client.0.smithi139.stdout: "perf_stat": { 2024-09-15T20:38:06.368 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ms": 100, 2024-09-15T20:38:06.369 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ms": 100, 2024-09-15T20:38:06.369 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ns": 100000000, 2024-09-15T20:38:06.369 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ns": 100000000 2024-09-15T20:38:06.369 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.369 INFO:tasks.workunit.client.0.smithi139.stdout: "alerts": [], 2024-09-15T20:38:06.369 INFO:tasks.workunit.client.0.smithi139.stdout: "network_ping_times": [] 2024-09-15T20:38:06.369 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.369 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.369 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_stats": [ 2024-09-15T20:38:06.369 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.369 INFO:tasks.workunit.client.0.smithi139.stdout: "poolid": 1, 2024-09-15T20:38:06.369 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg": 32, 2024-09-15T20:38:06.369 INFO:tasks.workunit.client.0.smithi139.stdout: "stat_sum": { 2024-09-15T20:38:06.370 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes": 60833, 2024-09-15T20:38:06.370 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects": 100, 2024-09-15T20:38:06.370 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_clones": 0, 2024-09-15T20:38:06.370 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_copies": 300, 2024-09-15T20:38:06.370 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T20:38:06.370 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing": 0, 2024-09-15T20:38:06.370 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_degraded": 100, 2024-09-15T20:38:06.370 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_misplaced": 0, 2024-09-15T20:38:06.370 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_unfound": 0, 2024-09-15T20:38:06.370 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_dirty": 100, 2024-09-15T20:38:06.370 INFO:tasks.workunit.client.0.smithi139.stdout: "num_whiteouts": 0, 2024-09-15T20:38:06.370 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read": 0, 2024-09-15T20:38:06.370 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read_kb": 0, 2024-09-15T20:38:06.370 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write": 58, 2024-09-15T20:38:06.371 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write_kb": 58, 2024-09-15T20:38:06.371 INFO:tasks.workunit.client.0.smithi139.stdout: "num_scrub_errors": 0, 2024-09-15T20:38:06.371 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T20:38:06.371 INFO:tasks.workunit.client.0.smithi139.stdout: "num_deep_scrub_errors": 0, 2024-09-15T20:38:06.371 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_recovered": 0, 2024-09-15T20:38:06.371 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_recovered": 0, 2024-09-15T20:38:06.371 INFO:tasks.workunit.client.0.smithi139.stdout: "num_keys_recovered": 0, 2024-09-15T20:38:06.371 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_omap": 0, 2024-09-15T20:38:06.371 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T20:38:06.371 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T20:38:06.371 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush": 0, 2024-09-15T20:38:06.371 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_kb": 0, 2024-09-15T20:38:06.371 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict": 0, 2024-09-15T20:38:06.372 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_kb": 0, 2024-09-15T20:38:06.372 INFO:tasks.workunit.client.0.smithi139.stdout: "num_promote": 0, 2024-09-15T20:38:06.372 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_high": 0, 2024-09-15T20:38:06.372 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_low": 0, 2024-09-15T20:38:06.372 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_some": 0, 2024-09-15T20:38:06.372 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_full": 0, 2024-09-15T20:38:06.372 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_pinned": 0, 2024-09-15T20:38:06.372 INFO:tasks.workunit.client.0.smithi139.stdout: "num_legacy_snapsets": 0, 2024-09-15T20:38:06.372 INFO:tasks.workunit.client.0.smithi139.stdout: "num_large_omap_objects": 0, 2024-09-15T20:38:06.372 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_manifest": 0, 2024-09-15T20:38:06.372 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_bytes": 0, 2024-09-15T20:38:06.372 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_keys": 0, 2024-09-15T20:38:06.372 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_repaired": 0 2024-09-15T20:38:06.373 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.373 INFO:tasks.workunit.client.0.smithi139.stdout: "store_stats": { 2024-09-15T20:38:06.373 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 0, 2024-09-15T20:38:06.373 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 0, 2024-09-15T20:38:06.373 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:38:06.373 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 1220608, 2024-09-15T20:38:06.373 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 184566, 2024-09-15T20:38:06.373 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:38:06.373 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:38:06.373 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:38:06.373 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 0, 2024-09-15T20:38:06.373 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 0 2024-09-15T20:38:06.373 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.373 INFO:tasks.workunit.client.0.smithi139.stdout: "log_size": 553, 2024-09-15T20:38:06.374 INFO:tasks.workunit.client.0.smithi139.stdout: "ondisk_log_size": 553, 2024-09-15T20:38:06.374 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 64, 2024-09-15T20:38:06.374 INFO:tasks.workunit.client.0.smithi139.stdout: "acting": 64, 2024-09-15T20:38:06.374 INFO:tasks.workunit.client.0.smithi139.stdout: "num_store_stats": 3 2024-09-15T20:38:06.374 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.374 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:06.374 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_stats": [ 2024-09-15T20:38:06.374 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.374 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:38:06.374 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 326417514508 2024-09-15T20:38:06.374 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.374 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.374 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:38:06.375 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 42949672992 2024-09-15T20:38:06.375 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.375 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.375 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:38:06.375 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 343597383690 2024-09-15T20:38:06.375 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.375 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:06.375 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_by_state": [ 2024-09-15T20:38:06.375 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.375 INFO:tasks.workunit.client.0.smithi139.stdout: "state": "active+undersized+degraded", 2024-09-15T20:38:06.375 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 32 2024-09-15T20:38:06.375 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.375 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:06.376 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_by_osd": [ 2024-09-15T20:38:06.376 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.376 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:38:06.376 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 16, 2024-09-15T20:38:06.376 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 32, 2024-09-15T20:38:06.376 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:38:06.376 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.376 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.376 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:38:06.376 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 16, 2024-09-15T20:38:06.376 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 32, 2024-09-15T20:38:06.376 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:38:06.376 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.377 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.377 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:38:06.377 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 0, 2024-09-15T20:38:06.377 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 0, 2024-09-15T20:38:06.377 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:38:06.377 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.377 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:06.377 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snaps": [ 2024-09-15T20:38:06.377 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:06.377 INFO:tasks.workunit.client.0.smithi139.stdout: "pool": 1, 2024-09-15T20:38:06.377 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snaps": {} 2024-09-15T20:38:06.377 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.377 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:06.377 INFO:tasks.workunit.client.0.smithi139.stdout: "servicemap": { 2024-09-15T20:38:06.378 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 5, 2024-09-15T20:38:06.378 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:37:09.885149+0000", 2024-09-15T20:38:06.378 INFO:tasks.workunit.client.0.smithi139.stdout: "services": {} 2024-09-15T20:38:06.378 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:06.378 INFO:tasks.workunit.client.0.smithi139.stdout: "mgrstat_first_committed": 1, 2024-09-15T20:38:06.378 INFO:tasks.workunit.client.0.smithi139.stdout: "mgrstat_last_committed": 101, 2024-09-15T20:38:06.378 INFO:tasks.workunit.client.0.smithi139.stdout: "logm_first_committed": 1, 2024-09-15T20:38:06.378 INFO:tasks.workunit.client.0.smithi139.stdout: "logm_last_committed": 474, 2024-09-15T20:38:06.378 INFO:tasks.workunit.client.0.smithi139.stdout: "paxos": { 2024-09-15T20:38:06.378 INFO:tasks.workunit.client.0.smithi139.stdout: "first_committed": 1, 2024-09-15T20:38:06.378 INFO:tasks.workunit.client.0.smithi139.stdout: "last_committed": 648, 2024-09-15T20:38:06.378 INFO:tasks.workunit.client.0.smithi139.stdout: "last_pn": 0, 2024-09-15T20:38:06.378 INFO:tasks.workunit.client.0.smithi139.stdout: "accepted_pn": 0 2024-09-15T20:38:06.378 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:06.379 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:38:06.379 INFO:tasks.workunit.client.0.smithi139.stderr:report 1708107713 2024-09-15T20:38:06.379 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-15T20:38:06.379 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: expr 2 + 2 2024-09-15T20:38:06.379 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: objname=existing_4 2024-09-15T20:38:06.380 INFO:tasks.workunit.client.0.smithi139.stdout:writing non-divergent object existing_4 2024-09-15T20:38:06.380 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2024-09-15T20:38:06.380 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2024-09-15T20:38:06.705 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:38:06.705 INFO:tasks.workunit.client.0.smithi139.stdout:1.1f 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394529+0000 41'29 87:103 [2,0] 2 [2,0] 2 37'28 2024-09-15T20:36:40.657710+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:26:32.119186+0000 2 0 2024-09-15T20:38:06.705 INFO:tasks.workunit.client.0.smithi139.stdout:1.1e 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394381+0000 46'29 87:116 [2,0] 2 [2,0] 2 40'28 2024-09-15T20:36:45.584734+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:06:07.931182+0000 2 0 2024-09-15T20:38:06.705 INFO:tasks.workunit.client.0.smithi139.stdout:1.1d 4 0 4 0 0 2468 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:29.424550+0000 42'31 87:114 [2,0] 2 [2,0] 2 36'29 2024-09-15T20:36:37.493943+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:34:28.615082+0000 2 0 2024-09-15T20:38:06.705 INFO:tasks.workunit.client.0.smithi139.stdout:1.b 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.396794+0000 45'29 87:115 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:44.549410+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:27:32.595447+0000 2 0 2024-09-15T20:38:06.706 INFO:tasks.workunit.client.0.smithi139.stdout:1.a 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.368887+0000 39'29 87:93 [2,0] 2 [2,0] 2 39'29 2024-09-15T20:36:56.804751+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:38:28.674672+0000 0 0 2024-09-15T20:38:06.706 INFO:tasks.workunit.client.0.smithi139.stdout:1.9 5 0 5 0 0 3085 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.368691+0000 45'29 87:118 [2,0] 2 [2,0] 2 45'29 2024-09-15T20:36:51.656322+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:14:38.500654+0000 3 0 2024-09-15T20:38:06.706 INFO:tasks.workunit.client.0.smithi139.stdout:1.8 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.499768+0000 45'30 87:106 [2,0] 2 [2,0] 2 45'30 2024-09-15T20:36:50.633607+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:06.706 INFO:tasks.workunit.client.0.smithi139.stdout:1.7 2 0 2 0 0 1138 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:29.424254+0000 38'28 87:80 [2,0] 2 [2,0] 2 38'28 2024-09-15T20:36:48.576911+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:06.706 INFO:tasks.workunit.client.0.smithi139.stdout:1.6 4 0 4 0 0 2468 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.410719+0000 42'30 87:102 [0,2] 0 [0,2] 0 42'30 2024-09-15T20:36:46.625811+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:02:22.839337+0000 2 0 2024-09-15T20:38:06.706 INFO:tasks.workunit.client.0.smithi139.stdout:1.5 6 0 6 0 0 3702 0 0 20 12 20 active+undersized+degraded 2024-09-15T20:37:28.369909+0000 45'32 87:127 [2,0] 2 [2,0] 2 40'30 2024-09-15T20:36:40.468115+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:22:28.237620+0000 4 0 2024-09-15T20:38:06.706 INFO:tasks.workunit.client.0.smithi139.stdout:1.4 2 0 2 0 0 1234 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.399006+0000 40'28 87:104 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:38.561548+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:24:41.193931+0000 1 0 2024-09-15T20:38:06.706 INFO:tasks.workunit.client.0.smithi139.stdout:1.3 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.046599+0000 47'30 87:108 [2,0] 2 [2,0] 2 47'30 2024-09-15T20:36:54.739486+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:06.706 INFO:tasks.workunit.client.0.smithi139.stdout:1.2 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.409088+0000 42'28 87:99 [0,2] 0 [0,2] 0 42'28 2024-09-15T20:36:44.289898+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:59:17.740071+0000 1 0 2024-09-15T20:38:06.706 INFO:tasks.workunit.client.0.smithi139.stdout:1.c 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.395833+0000 44'29 87:116 [0,2] 0 [0,2] 0 38'28 2024-09-15T20:36:43.561129+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:59:06.361170+0000 2 0 2024-09-15T20:38:06.706 INFO:tasks.workunit.client.0.smithi139.stdout:1.1 2 0 2 0 0 1234 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:29.499685+0000 44'29 87:104 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:42.549754+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:03:26.939083+0000 0 0 2024-09-15T20:38:06.706 INFO:tasks.workunit.client.0.smithi139.stdout:1.1c 2 0 2 0 0 1234 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.398522+0000 36'29 87:95 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:34.684895+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:11:58.732534+0000 1 0 2024-09-15T20:38:06.706 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.409170+0000 45'29 87:119 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:31.575682+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:19:06.387325+0000 1 0 2024-09-15T20:38:06.707 INFO:tasks.workunit.client.0.smithi139.stdout:1.d 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394417+0000 45'29 87:113 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:39.548519+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:37:36.633002+0000 1 0 2024-09-15T20:38:06.707 INFO:tasks.workunit.client.0.smithi139.stdout:1.e 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:29.097547+0000 44'29 87:93 [2,0] 2 [2,0] 2 44'29 2024-09-15T20:36:49.578000+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:06.707 INFO:tasks.workunit.client.0.smithi139.stdout:1.f 4 0 4 0 0 2468 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.396239+0000 41'29 87:103 [2,0] 2 [2,0] 2 34'28 2024-09-15T20:36:32.588850+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:09:31.037907+0000 3 0 2024-09-15T20:38:06.707 INFO:tasks.workunit.client.0.smithi139.stdout:1.10 3 0 3 0 0 1851 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.397764+0000 45'30 87:120 [0,2] 0 [0,2] 0 38'29 2024-09-15T20:36:39.578100+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:37:31.289109+0000 2 0 2024-09-15T20:38:06.707 INFO:tasks.workunit.client.0.smithi139.stdout:1.11 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.398513+0000 37'29 87:90 [0,2] 0 [0,2] 0 36'28 2024-09-15T20:36:36.444590+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:48:59.386094+0000 2 0 2024-09-15T20:38:06.707 INFO:tasks.workunit.client.0.smithi139.stdout:1.12 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.397545+0000 36'28 87:105 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:35.623615+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:48:18.440517+0000 0 0 2024-09-15T20:38:06.707 INFO:tasks.workunit.client.0.smithi139.stdout:1.13 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.396874+0000 45'30 87:108 [0,2] 0 [0,2] 0 45'30 2024-09-15T20:36:55.766014+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:38:28.994911+0000 0 0 2024-09-15T20:38:06.707 INFO:tasks.workunit.client.0.smithi139.stdout:1.14 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:29.122393+0000 45'29 87:114 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:37.554032+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:59:56.311369+0000 1 0 2024-09-15T20:38:06.707 INFO:tasks.workunit.client.0.smithi139.stdout:1.15 3 0 3 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.411771+0000 41'29 87:105 [0,2] 0 [0,2] 0 41'29 2024-09-15T20:36:41.506894+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:06.707 INFO:tasks.workunit.client.0.smithi139.stdout:1.16 4 0 4 0 0 2468 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:28.410181+0000 41'31 87:115 [0,2] 0 [0,2] 0 41'31 2024-09-15T20:36:47.576518+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:29:37.611785+0000 3 0 2024-09-15T20:38:06.708 INFO:tasks.workunit.client.0.smithi139.stdout:1.17 1 0 1 0 0 520 0 0 15 12 15 active+undersized+degraded 2024-09-15T20:37:28.409266+0000 32'27 87:68 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:42.639678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:06.708 INFO:tasks.workunit.client.0.smithi139.stdout:1.18 3 0 3 0 0 1851 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.394810+0000 45'30 87:119 [2,0] 2 [2,0] 2 34'28 2024-09-15T20:36:36.622678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:29:24.808950+0000 1 0 2024-09-15T20:38:06.708 INFO:tasks.workunit.client.0.smithi139.stdout:1.19 2 0 2 0 0 1137 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.398489+0000 34'28 87:81 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:52.759747+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:06.708 INFO:tasks.workunit.client.0.smithi139.stdout:1.1a 4 0 4 0 0 2468 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.369086+0000 44'30 87:116 [2,0] 2 [2,0] 2 44'30 2024-09-15T20:36:53.716793+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:57:48.429965+0000 3 0 2024-09-15T20:38:06.708 INFO:tasks.workunit.client.0.smithi139.stdout:1.1b 5 0 5 0 0 3085 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:28.396910+0000 45'31 87:124 [0,2] 0 [0,2] 0 41'30 2024-09-15T20:36:41.658323+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:57:15.784191+0000 4 0 2024-09-15T20:38:06.708 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:38:06.708 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:38:06.708 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:38:06.720 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2024-09-15T20:38:06.818 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T20:38:06.818 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: wait_for_clean 2024-09-15T20:38:06.818 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:38:06.818 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:38:06.818 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:38:06.819 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T20:38:06.819 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:38:06.819 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:38:06.820 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:38:06.820 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:38:06.820 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:38:06.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T20:38:06.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:38:06.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:38:06.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:38:06.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:38:06.889 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:38:07.360 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:38:07.361 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:38:07.361 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:38:07.361 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:38:07.361 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:38:07.361 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:38:07.536 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=326417514510 2024-09-15T20:38:07.536 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 326417514510 2024-09-15T20:38:07.537 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-326417514510' 2024-09-15T20:38:07.537 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:38:07.537 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:38:07.665 INFO:tasks.workunit.client.0.smithi139.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-15T20:38:07.667 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-15T20:38:07.668 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-15T20:38:07.668 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-15T20:38:07.668 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:38:07.668 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:38:07.817 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383693 2024-09-15T20:38:07.817 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383693 2024-09-15T20:38:07.818 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-326417514510 2-343597383693' 2024-09-15T20:38:07.818 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:38:07.818 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-326417514510 2024-09-15T20:38:07.818 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:38:07.819 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:38:07.819 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-326417514510 2024-09-15T20:38:07.819 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:38:07.820 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=326417514510 2024-09-15T20:38:07.820 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 326417514510 2024-09-15T20:38:07.821 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 326417514510' 2024-09-15T20:38:07.821 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:38:08.236 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 326417514510 -lt 326417514510 2024-09-15T20:38:08.236 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:38:08.237 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-343597383693 2024-09-15T20:38:08.237 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:38:08.238 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:38:08.239 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-343597383693 2024-09-15T20:38:08.239 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:38:08.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383693 2024-09-15T20:38:08.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 343597383693' 2024-09-15T20:38:08.241 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 343597383693 2024-09-15T20:38:08.241 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:38:08.667 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383693 -lt 343597383693 2024-09-15T20:38:08.668 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:38:08.669 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:38:08.669 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:38:09.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-15T20:38:09.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:38:09.312 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:38:09.312 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:38:09.312 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:38:09.312 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:38:09.312 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:38:09.312 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:38:09.738 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:38:09.739 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:38:09.739 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:38:09.739 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:38:10.291 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-15T20:38:10.291 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T20:38:10.292 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:38:10.292 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T20:38:10.292 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:38:10.292 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:38:10.392 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:38:10.393 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:38:10.393 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:38:10.394 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:38:10.394 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:38:10.394 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:38:10.394 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:38:10.394 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:38:10.837 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:38:10.838 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:38:10.838 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:38:10.838 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:38:11.391 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-15T20:38:11.392 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:38:11.392 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:38:11.392 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:38:11.392 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:38:11.392 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:38:11.392 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:38:11.392 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:38:11.392 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:38:11.956 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:38:11.956 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:38:11.956 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-15T20:38:11.956 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:38:11.956 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:38:12.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:38:12.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:38:12.158 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:38:12.158 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:38:12.158 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:38:12.158 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:38:12.159 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:38:12.159 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:38:12.648 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:38:12.649 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:38:12.649 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:38:12.649 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:38:13.194 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-15T20:38:13.194 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:38:13.194 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:38:13.194 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:38:13.194 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:38:13.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:38:13.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:38:13.195 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:38:13.195 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:38:13.750 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:38:13.750 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:38:13.750 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-15T20:38:13.750 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:38:13.750 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T20:38:14.151 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:38:14.151 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:38:14.152 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:38:14.152 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:38:14.152 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:38:14.153 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:38:14.153 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:38:14.153 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:38:14.598 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:38:14.598 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:38:14.599 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:38:14.599 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:38:15.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-15T20:38:15.170 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:38:15.170 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:38:15.170 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:38:15.170 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:38:15.170 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:38:15.170 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:38:15.170 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:38:15.170 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:38:15.738 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:38:15.739 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:38:15.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-15T20:38:15.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:38:15.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-15T20:38:16.539 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:38:16.539 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:38:16.540 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:38:16.541 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:38:16.541 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:38:16.541 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:38:16.541 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:38:16.541 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:38:16.989 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:38:16.989 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:38:16.990 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:38:16.990 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:38:17.562 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-15T20:38:17.562 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:38:17.562 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:38:17.562 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:38:17.562 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:38:17.562 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:38:17.562 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:38:17.563 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:38:17.563 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:38:18.149 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:38:18.149 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:38:18.150 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-15T20:38:18.150 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:38:18.150 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-15T20:38:19.751 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:38:19.751 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:38:19.752 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:38:19.752 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:38:19.752 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:38:19.752 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:38:19.752 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:38:19.753 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:38:20.182 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:38:20.182 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:38:20.183 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:38:20.183 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:38:20.821 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-15T20:38:20.821 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:38:20.822 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:38:20.822 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:38:20.822 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:38:20.822 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:38:20.822 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:38:20.822 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:38:20.822 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:38:21.374 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:38:21.374 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:38:21.374 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-15T20:38:21.375 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:38:21.375 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-15T20:38:24.576 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:38:24.576 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:38:24.577 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:38:24.577 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:38:24.577 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:38:24.577 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:38:24.578 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:38:24.578 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:38:25.015 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:38:25.016 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:38:25.016 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:38:25.016 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:38:25.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-15T20:38:25.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:38:25.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:38:25.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:38:25.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:38:25.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:38:25.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:38:25.565 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:38:25.565 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:38:26.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:38:26.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:38:26.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-15T20:38:26.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:38:26.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-15T20:38:32.529 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:38:32.529 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:38:32.530 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:38:32.530 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:38:32.530 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:38:32.530 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:38:32.531 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:38:32.531 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:38:32.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:38:32.978 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:38:32.978 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:38:32.978 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:38:33.541 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-15T20:38:33.542 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:38:33.542 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:38:33.542 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:38:33.542 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:38:33.542 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:38:33.542 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:38:33.542 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:38:33.543 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:38:34.097 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:38:34.097 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:38:34.097 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-15T20:38:34.097 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:38:34.097 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-15T20:38:41.398 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:38:41.398 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:38:41.399 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:38:41.399 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:38:41.399 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:38:41.399 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:38:41.400 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:38:41.400 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:38:41.854 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:38:41.855 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:38:41.855 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:38:41.856 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:38:42.422 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-15T20:38:42.423 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:38:42.423 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:38:42.423 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:38:42.423 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:38:42.423 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:38:42.423 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:38:42.423 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:38:42.423 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:38:42.984 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:38:42.984 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:38:42.984 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-15T20:38:42.984 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-15T20:38:43.543 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:38:43.543 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_fingerprint": "f912e059-3564-4d24-9665-24ea498f0830", 2024-09-15T20:38:43.543 INFO:tasks.workunit.client.0.smithi139.stdout: "version": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:38:43.543 INFO:tasks.workunit.client.0.smithi139.stdout: "commit": "fc3e0bff7468fd55478adbae7f75508b749124b6", 2024-09-15T20:38:43.543 INFO:tasks.workunit.client.0.smithi139.stdout: "timestamp": "2024-09-15T20:38:43.546455+0000", 2024-09-15T20:38:43.543 INFO:tasks.workunit.client.0.smithi139.stdout: "tag": "", 2024-09-15T20:38:43.543 INFO:tasks.workunit.client.0.smithi139.stdout: "health": { 2024-09-15T20:38:43.544 INFO:tasks.workunit.client.0.smithi139.stdout: "status": "HEALTH_WARN", 2024-09-15T20:38:43.544 INFO:tasks.workunit.client.0.smithi139.stdout: "checks": { 2024-09-15T20:38:43.544 INFO:tasks.workunit.client.0.smithi139.stdout: "OSDMAP_FLAGS": { 2024-09-15T20:38:43.544 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:38:43.544 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:38:43.544 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-15T20:38:43.544 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 17 2024-09-15T20:38:43.544 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.544 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [], 2024-09-15T20:38:43.544 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:38:43.545 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.545 INFO:tasks.workunit.client.0.smithi139.stdout: "OSD_DOWN": { 2024-09-15T20:38:43.545 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:38:43.545 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:38:43.545 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "1 osds down", 2024-09-15T20:38:43.545 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 1 2024-09-15T20:38:43.545 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.545 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:38:43.545 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.545 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "osd.1 (root=default,host=smithi139) is down" 2024-09-15T20:38:43.546 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.546 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:43.546 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:38:43.546 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.546 INFO:tasks.workunit.client.0.smithi139.stdout: "PG_DEGRADED": { 2024-09-15T20:38:43.546 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:38:43.546 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:38:43.546 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2024-09-15T20:38:43.546 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 64 2024-09-15T20:38:43.546 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.546 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:38:43.546 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.546 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.0 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T20:38:43.546 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.547 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.547 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.1 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T20:38:43.547 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.547 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.547 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.2 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T20:38:43.547 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.547 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.547 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.3 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T20:38:43.547 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.547 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.547 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.4 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T20:38:43.547 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.547 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.548 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.5 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T20:38:43.548 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.548 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.548 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.6 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T20:38:43.548 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.548 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.548 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.7 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T20:38:43.548 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.548 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.548 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.8 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T20:38:43.548 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.548 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.548 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.9 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T20:38:43.549 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.549 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.549 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.a is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T20:38:43.549 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.549 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.549 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.b is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T20:38:43.549 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.549 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.549 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.c is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T20:38:43.549 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.549 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.549 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.d is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T20:38:43.549 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.549 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.550 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.e is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T20:38:43.550 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.550 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.550 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.f is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T20:38:43.550 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.550 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.550 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.10 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T20:38:43.550 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.550 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.550 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.11 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T20:38:43.551 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.551 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.551 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.12 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T20:38:43.551 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.551 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.551 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.13 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T20:38:43.551 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.551 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.551 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.14 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T20:38:43.551 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.551 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.551 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.15 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T20:38:43.551 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.551 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.552 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.16 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T20:38:43.552 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.552 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.552 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.17 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T20:38:43.552 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.552 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.552 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.18 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T20:38:43.552 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.552 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.552 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.19 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T20:38:43.552 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.552 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.553 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.1a is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T20:38:43.553 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.553 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.553 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.1b is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T20:38:43.553 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.553 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.553 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.1c is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T20:38:43.553 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.553 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.554 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.1d is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T20:38:43.554 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.554 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.554 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.1e is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T20:38:43.554 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.554 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.554 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.1f is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T20:38:43.554 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.554 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:43.554 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:38:43.554 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.554 INFO:tasks.workunit.client.0.smithi139.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-15T20:38:43.555 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:38:43.555 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:38:43.555 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-15T20:38:43.555 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 1 2024-09-15T20:38:43.555 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.555 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:38:43.555 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.555 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "application not enabled on pool 'test'" 2024-09-15T20:38:43.555 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.555 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.555 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-15T20:38:43.555 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.555 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:43.555 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:38:43.556 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.556 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.556 INFO:tasks.workunit.client.0.smithi139.stdout: "mutes": [] 2024-09-15T20:38:43.556 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.556 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap_first_committed": 1, 2024-09-15T20:38:43.556 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap_last_committed": 1, 2024-09-15T20:38:43.556 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap": { 2024-09-15T20:38:43.556 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 1, 2024-09-15T20:38:43.556 INFO:tasks.workunit.client.0.smithi139.stdout: "fsid": "0e220df1-ac9c-433e-8885-cf9f7aab293e", 2024-09-15T20:38:43.556 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:34:21.257147Z", 2024-09-15T20:38:43.556 INFO:tasks.workunit.client.0.smithi139.stdout: "created": "2024-09-15T20:34:21.257147Z", 2024-09-15T20:38:43.556 INFO:tasks.workunit.client.0.smithi139.stdout: "min_mon_release": 19, 2024-09-15T20:38:43.556 INFO:tasks.workunit.client.0.smithi139.stdout: "min_mon_release_name": "squid", 2024-09-15T20:38:43.557 INFO:tasks.workunit.client.0.smithi139.stdout: "election_strategy": 1, 2024-09-15T20:38:43.557 INFO:tasks.workunit.client.0.smithi139.stdout: "disallowed_leaders: ": "", 2024-09-15T20:38:43.557 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode": false, 2024-09-15T20:38:43.557 INFO:tasks.workunit.client.0.smithi139.stdout: "tiebreaker_mon": "", 2024-09-15T20:38:43.557 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_ranks: ": "", 2024-09-15T20:38:43.557 INFO:tasks.workunit.client.0.smithi139.stdout: "features": { 2024-09-15T20:38:43.557 INFO:tasks.workunit.client.0.smithi139.stdout: "persistent": [ 2024-09-15T20:38:43.557 INFO:tasks.workunit.client.0.smithi139.stdout: "kraken", 2024-09-15T20:38:43.557 INFO:tasks.workunit.client.0.smithi139.stdout: "luminous", 2024-09-15T20:38:43.557 INFO:tasks.workunit.client.0.smithi139.stdout: "mimic", 2024-09-15T20:38:43.557 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap-prune", 2024-09-15T20:38:43.557 INFO:tasks.workunit.client.0.smithi139.stdout: "nautilus", 2024-09-15T20:38:43.557 INFO:tasks.workunit.client.0.smithi139.stdout: "octopus", 2024-09-15T20:38:43.558 INFO:tasks.workunit.client.0.smithi139.stdout: "pacific", 2024-09-15T20:38:43.558 INFO:tasks.workunit.client.0.smithi139.stdout: "elector-pinging", 2024-09-15T20:38:43.558 INFO:tasks.workunit.client.0.smithi139.stdout: "quincy", 2024-09-15T20:38:43.558 INFO:tasks.workunit.client.0.smithi139.stdout: "reef", 2024-09-15T20:38:43.558 INFO:tasks.workunit.client.0.smithi139.stdout: "squid" 2024-09-15T20:38:43.558 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:43.558 INFO:tasks.workunit.client.0.smithi139.stdout: "optional": [] 2024-09-15T20:38:43.558 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.558 INFO:tasks.workunit.client.0.smithi139.stdout: "mons": [ 2024-09-15T20:38:43.558 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.558 INFO:tasks.workunit.client.0.smithi139.stdout: "rank": 0, 2024-09-15T20:38:43.559 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "a", 2024-09-15T20:38:43.559 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:38:43.559 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:43.559 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.559 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:43.559 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:7115", 2024-09-15T20:38:43.559 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 0 2024-09-15T20:38:43.559 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.559 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.559 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.559 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:7115/0", 2024-09-15T20:38:43.560 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-15T20:38:43.560 INFO:tasks.workunit.client.0.smithi139.stdout: "priority": 0, 2024-09-15T20:38:43.560 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 0, 2024-09-15T20:38:43.560 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_location": "{}" 2024-09-15T20:38:43.560 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.560 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.560 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.560 INFO:tasks.workunit.client.0.smithi139.stdout: "quorum": [ 2024-09-15T20:38:43.560 INFO:tasks.workunit.client.0.smithi139.stdout: 0 2024-09-15T20:38:43.560 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:43.560 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap": { 2024-09-15T20:38:43.560 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 87, 2024-09-15T20:38:43.560 INFO:tasks.workunit.client.0.smithi139.stdout: "fsid": "0e220df1-ac9c-433e-8885-cf9f7aab293e", 2024-09-15T20:38:43.561 INFO:tasks.workunit.client.0.smithi139.stdout: "created": "2024-09-15T20:34:22.232759+0000", 2024-09-15T20:38:43.561 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:37:28.945712+0000", 2024-09-15T20:38:43.561 INFO:tasks.workunit.client.0.smithi139.stdout: "last_up_change": "2024-09-15T20:37:26.014870+0000", 2024-09-15T20:38:43.561 INFO:tasks.workunit.client.0.smithi139.stdout: "last_in_change": "2024-09-15T20:37:27.456159+0000", 2024-09-15T20:38:43.561 INFO:tasks.workunit.client.0.smithi139.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T20:38:43.561 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_num": 5800384, 2024-09-15T20:38:43.561 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_set": [ 2024-09-15T20:38:43.561 INFO:tasks.workunit.client.0.smithi139.stdout: "nodown", 2024-09-15T20:38:43.561 INFO:tasks.workunit.client.0.smithi139.stdout: "noin", 2024-09-15T20:38:43.561 INFO:tasks.workunit.client.0.smithi139.stdout: "noout", 2024-09-15T20:38:43.561 INFO:tasks.workunit.client.0.smithi139.stdout: "pglog_hardlimit", 2024-09-15T20:38:43.561 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snapdirs", 2024-09-15T20:38:43.561 INFO:tasks.workunit.client.0.smithi139.stdout: "recovery_deletes", 2024-09-15T20:38:43.561 INFO:tasks.workunit.client.0.smithi139.stdout: "sortbitwise" 2024-09-15T20:38:43.562 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:43.562 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_version": 9, 2024-09-15T20:38:43.562 INFO:tasks.workunit.client.0.smithi139.stdout: "full_ratio": 0.99000000953674316, 2024-09-15T20:38:43.562 INFO:tasks.workunit.client.0.smithi139.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-15T20:38:43.562 INFO:tasks.workunit.client.0.smithi139.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T20:38:43.562 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_snapshot": "", 2024-09-15T20:38:43.562 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_max": 1, 2024-09-15T20:38:43.562 INFO:tasks.workunit.client.0.smithi139.stdout: "max_osd": 3, 2024-09-15T20:38:43.562 INFO:tasks.workunit.client.0.smithi139.stdout: "require_min_compat_client": "luminous", 2024-09-15T20:38:43.562 INFO:tasks.workunit.client.0.smithi139.stdout: "min_compat_client": "jewel", 2024-09-15T20:38:43.562 INFO:tasks.workunit.client.0.smithi139.stdout: "require_osd_release": "squid", 2024-09-15T20:38:43.562 INFO:tasks.workunit.client.0.smithi139.stdout: "allow_crimson": false, 2024-09-15T20:38:43.562 INFO:tasks.workunit.client.0.smithi139.stdout: "pools": [ 2024-09-15T20:38:43.562 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.563 INFO:tasks.workunit.client.0.smithi139.stdout: "pool": 1, 2024-09-15T20:38:43.563 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_name": "test", 2024-09-15T20:38:43.563 INFO:tasks.workunit.client.0.smithi139.stdout: "create_time": "2024-09-15T20:36:11.566515+0000", 2024-09-15T20:38:43.563 INFO:tasks.workunit.client.0.smithi139.stdout: "flags": 1, 2024-09-15T20:38:43.563 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_names": "hashpspool", 2024-09-15T20:38:43.563 INFO:tasks.workunit.client.0.smithi139.stdout: "type": 1, 2024-09-15T20:38:43.563 INFO:tasks.workunit.client.0.smithi139.stdout: "size": 3, 2024-09-15T20:38:43.563 INFO:tasks.workunit.client.0.smithi139.stdout: "min_size": 2, 2024-09-15T20:38:43.563 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_rule": 0, 2024-09-15T20:38:43.563 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_count": 0, 2024-09-15T20:38:43.563 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_target": 0, 2024-09-15T20:38:43.563 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T20:38:43.563 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T20:38:43.563 INFO:tasks.workunit.client.0.smithi139.stdout: "is_stretch_pool": false, 2024-09-15T20:38:43.564 INFO:tasks.workunit.client.0.smithi139.stdout: "object_hash": 2, 2024-09-15T20:38:43.564 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_autoscale_mode": "on", 2024-09-15T20:38:43.564 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num": 32, 2024-09-15T20:38:43.564 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_placement_num": 27, 2024-09-15T20:38:43.564 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_placement_num_target": 32, 2024-09-15T20:38:43.564 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num_target": 32, 2024-09-15T20:38:43.564 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num_pending": 32, 2024-09-15T20:38:43.564 INFO:tasks.workunit.client.0.smithi139.stdout: "last_pg_merge_meta": { 2024-09-15T20:38:43.564 INFO:tasks.workunit.client.0.smithi139.stdout: "source_pgid": "0.0", 2024-09-15T20:38:43.564 INFO:tasks.workunit.client.0.smithi139.stdout: "ready_epoch": 0, 2024-09-15T20:38:43.564 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_started": 0, 2024-09-15T20:38:43.564 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_clean": 0, 2024-09-15T20:38:43.564 INFO:tasks.workunit.client.0.smithi139.stdout: "source_version": "0'0", 2024-09-15T20:38:43.564 INFO:tasks.workunit.client.0.smithi139.stdout: "target_version": "0'0" 2024-09-15T20:38:43.565 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.565 INFO:tasks.workunit.client.0.smithi139.stdout: "last_change": "84", 2024-09-15T20:38:43.565 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend": "0", 2024-09-15T20:38:43.565 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T20:38:43.565 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend_preluminous": "33", 2024-09-15T20:38:43.565 INFO:tasks.workunit.client.0.smithi139.stdout: "auid": 0, 2024-09-15T20:38:43.565 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_mode": "selfmanaged", 2024-09-15T20:38:43.565 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_seq": 0, 2024-09-15T20:38:43.565 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_epoch": 0, 2024-09-15T20:38:43.565 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_snaps": [], 2024-09-15T20:38:43.565 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_snaps": "[]", 2024-09-15T20:38:43.565 INFO:tasks.workunit.client.0.smithi139.stdout: "quota_max_bytes": 0, 2024-09-15T20:38:43.565 INFO:tasks.workunit.client.0.smithi139.stdout: "quota_max_objects": 0, 2024-09-15T20:38:43.566 INFO:tasks.workunit.client.0.smithi139.stdout: "tiers": [], 2024-09-15T20:38:43.566 INFO:tasks.workunit.client.0.smithi139.stdout: "tier_of": -1, 2024-09-15T20:38:43.566 INFO:tasks.workunit.client.0.smithi139.stdout: "read_tier": -1, 2024-09-15T20:38:43.566 INFO:tasks.workunit.client.0.smithi139.stdout: "write_tier": -1, 2024-09-15T20:38:43.566 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_mode": "none", 2024-09-15T20:38:43.566 INFO:tasks.workunit.client.0.smithi139.stdout: "target_max_bytes": 0, 2024-09-15T20:38:43.566 INFO:tasks.workunit.client.0.smithi139.stdout: "target_max_objects": 0, 2024-09-15T20:38:43.566 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T20:38:43.566 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T20:38:43.566 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T20:38:43.566 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_min_flush_age": 0, 2024-09-15T20:38:43.566 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_min_evict_age": 0, 2024-09-15T20:38:43.566 INFO:tasks.workunit.client.0.smithi139.stdout: "erasure_code_profile": "", 2024-09-15T20:38:43.566 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_params": { 2024-09-15T20:38:43.567 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "none" 2024-09-15T20:38:43.567 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.567 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_period": 0, 2024-09-15T20:38:43.567 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_count": 0, 2024-09-15T20:38:43.567 INFO:tasks.workunit.client.0.smithi139.stdout: "use_gmt_hitset": true, 2024-09-15T20:38:43.567 INFO:tasks.workunit.client.0.smithi139.stdout: "min_read_recency_for_promote": 0, 2024-09-15T20:38:43.567 INFO:tasks.workunit.client.0.smithi139.stdout: "min_write_recency_for_promote": 0, 2024-09-15T20:38:43.567 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T20:38:43.567 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_search_last_n": 0, 2024-09-15T20:38:43.567 INFO:tasks.workunit.client.0.smithi139.stdout: "grade_table": [], 2024-09-15T20:38:43.568 INFO:tasks.workunit.client.0.smithi139.stdout: "stripe_width": 0, 2024-09-15T20:38:43.568 INFO:tasks.workunit.client.0.smithi139.stdout: "expected_num_objects": 0, 2024-09-15T20:38:43.568 INFO:tasks.workunit.client.0.smithi139.stdout: "fast_read": false, 2024-09-15T20:38:43.568 INFO:tasks.workunit.client.0.smithi139.stdout: "options": {}, 2024-09-15T20:38:43.568 INFO:tasks.workunit.client.0.smithi139.stdout: "application_metadata": {}, 2024-09-15T20:38:43.568 INFO:tasks.workunit.client.0.smithi139.stdout: "read_balance": { 2024-09-15T20:38:43.568 INFO:tasks.workunit.client.0.smithi139.stdout: "score_type": "Fair distribution", 2024-09-15T20:38:43.568 INFO:tasks.workunit.client.0.smithi139.stdout: "score_acting": 1.4900000095367432, 2024-09-15T20:38:43.568 INFO:tasks.workunit.client.0.smithi139.stdout: "score_stable": 1.4900000095367432, 2024-09-15T20:38:43.568 INFO:tasks.workunit.client.0.smithi139.stdout: "optimal_score": 0.67000001668930054, 2024-09-15T20:38:43.568 INFO:tasks.workunit.client.0.smithi139.stdout: "raw_score_acting": 1, 2024-09-15T20:38:43.568 INFO:tasks.workunit.client.0.smithi139.stdout: "raw_score_stable": 1, 2024-09-15T20:38:43.568 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity_weighted": 1, 2024-09-15T20:38:43.568 INFO:tasks.workunit.client.0.smithi139.stdout: "average_primary_affinity": 1, 2024-09-15T20:38:43.569 INFO:tasks.workunit.client.0.smithi139.stdout: "average_primary_affinity_weighted": 1 2024-09-15T20:38:43.569 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.569 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.569 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:43.569 INFO:tasks.workunit.client.0.smithi139.stdout: "osds": [ 2024-09-15T20:38:43.569 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.569 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:38:43.569 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "8ceaad64-e87a-460e-8e98-3caf0f7eff12", 2024-09-15T20:38:43.569 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 1, 2024-09-15T20:38:43.569 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:38:43.569 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:38:43.570 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:38:43.570 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 5, 2024-09-15T20:38:43.570 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 54, 2024-09-15T20:38:43.570 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 76, 2024-09-15T20:38:43.570 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 83, 2024-09-15T20:38:43.570 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 66, 2024-09-15T20:38:43.570 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:38:43.570 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:38:43.570 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:43.570 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.570 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:43.570 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6802", 2024-09-15T20:38:43.570 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3684664553 2024-09-15T20:38:43.571 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.571 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.571 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:43.571 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6803", 2024-09-15T20:38:43.571 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3684664553 2024-09-15T20:38:43.571 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.571 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.571 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.571 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:38:43.571 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:43.571 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.571 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:43.571 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6804", 2024-09-15T20:38:43.571 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3684664553 2024-09-15T20:38:43.571 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.572 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.572 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:43.572 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6805", 2024-09-15T20:38:43.572 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3684664553 2024-09-15T20:38:43.572 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.572 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.572 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.572 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:38:43.572 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:43.572 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.572 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:43.572 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6808", 2024-09-15T20:38:43.572 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3684664553 2024-09-15T20:38:43.573 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.573 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.573 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:43.573 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6809", 2024-09-15T20:38:43.573 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3684664553 2024-09-15T20:38:43.573 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.573 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.573 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.573 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:38:43.573 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:43.573 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.573 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:43.573 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6806", 2024-09-15T20:38:43.573 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3684664553 2024-09-15T20:38:43.574 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.574 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.574 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:43.574 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6807", 2024-09-15T20:38:43.574 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3684664553 2024-09-15T20:38:43.574 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.574 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.574 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.574 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6803/3684664553", 2024-09-15T20:38:43.574 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6805/3684664553", 2024-09-15T20:38:43.574 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3684664553", 2024-09-15T20:38:43.575 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3684664553", 2024-09-15T20:38:43.575 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:38:43.575 INFO:tasks.workunit.client.0.smithi139.stdout: "exists", 2024-09-15T20:38:43.575 INFO:tasks.workunit.client.0.smithi139.stdout: "up" 2024-09-15T20:38:43.575 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.575 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.575 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.575 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:38:43.575 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "d5b37d03-d9a9-4f3f-8c3e-d61cdbdce38e", 2024-09-15T20:38:43.575 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 0, 2024-09-15T20:38:43.575 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:38:43.575 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:38:43.575 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:38:43.576 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 0, 2024-09-15T20:38:43.576 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 0, 2024-09-15T20:38:43.576 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 10, 2024-09-15T20:38:43.576 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 57, 2024-09-15T20:38:43.576 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 67, 2024-09-15T20:38:43.576 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:38:43.576 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:38:43.576 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:43.576 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.576 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:43.576 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6810", 2024-09-15T20:38:43.576 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 351107530 2024-09-15T20:38:43.576 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.576 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.577 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:43.577 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6811", 2024-09-15T20:38:43.577 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 351107530 2024-09-15T20:38:43.577 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.577 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.577 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.577 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:38:43.577 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:43.577 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.577 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:43.577 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6812", 2024-09-15T20:38:43.577 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 351107530 2024-09-15T20:38:43.577 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.578 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.578 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:43.578 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6813", 2024-09-15T20:38:43.578 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 351107530 2024-09-15T20:38:43.578 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.578 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.578 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.578 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:38:43.578 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:43.578 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.578 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:43.578 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6816", 2024-09-15T20:38:43.578 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 351107530 2024-09-15T20:38:43.578 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.579 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.579 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:43.579 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6817", 2024-09-15T20:38:43.579 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 351107530 2024-09-15T20:38:43.579 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.579 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.579 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.579 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:38:43.579 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:43.579 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.579 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:43.579 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6814", 2024-09-15T20:38:43.579 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 351107530 2024-09-15T20:38:43.579 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.580 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.580 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:43.580 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6815", 2024-09-15T20:38:43.580 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 351107530 2024-09-15T20:38:43.580 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.580 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.580 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.580 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6811/351107530", 2024-09-15T20:38:43.580 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6813/351107530", 2024-09-15T20:38:43.580 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6817/351107530", 2024-09-15T20:38:43.580 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6815/351107530", 2024-09-15T20:38:43.580 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:38:43.580 INFO:tasks.workunit.client.0.smithi139.stdout: "exists" 2024-09-15T20:38:43.580 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.581 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.581 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.581 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:38:43.581 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "5347fe2a-12cd-44fe-9cd9-0dbc2bc7ebb8", 2024-09-15T20:38:43.581 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 1, 2024-09-15T20:38:43.581 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:38:43.581 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:38:43.581 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:38:43.581 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 15, 2024-09-15T20:38:43.581 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 53, 2024-09-15T20:38:43.581 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 80, 2024-09-15T20:38:43.581 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 85, 2024-09-15T20:38:43.581 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 68, 2024-09-15T20:38:43.581 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:38:43.582 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:38:43.582 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:43.582 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.582 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:43.582 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6810", 2024-09-15T20:38:43.582 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3676612316 2024-09-15T20:38:43.582 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.582 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.582 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:43.582 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6811", 2024-09-15T20:38:43.582 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3676612316 2024-09-15T20:38:43.582 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.582 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.582 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.583 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:38:43.583 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:43.583 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.583 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:43.583 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6812", 2024-09-15T20:38:43.583 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3676612316 2024-09-15T20:38:43.583 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.583 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.583 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:43.583 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6813", 2024-09-15T20:38:43.583 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3676612316 2024-09-15T20:38:43.583 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.583 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.584 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.584 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:38:43.584 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:43.584 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.584 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:43.584 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6816", 2024-09-15T20:38:43.584 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3676612316 2024-09-15T20:38:43.584 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.584 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.584 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:43.584 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6817", 2024-09-15T20:38:43.584 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3676612316 2024-09-15T20:38:43.584 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.584 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.585 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.585 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:38:43.585 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:38:43.585 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.585 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:38:43.585 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6814", 2024-09-15T20:38:43.585 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3676612316 2024-09-15T20:38:43.585 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.585 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.585 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:38:43.585 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6815", 2024-09-15T20:38:43.585 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3676612316 2024-09-15T20:38:43.585 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.585 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.585 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.586 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6811/3676612316", 2024-09-15T20:38:43.586 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6813/3676612316", 2024-09-15T20:38:43.586 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3676612316", 2024-09-15T20:38:43.586 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3676612316", 2024-09-15T20:38:43.586 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:38:43.586 INFO:tasks.workunit.client.0.smithi139.stdout: "exists", 2024-09-15T20:38:43.586 INFO:tasks.workunit.client.0.smithi139.stdout: "up" 2024-09-15T20:38:43.586 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.586 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.586 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:43.586 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_xinfo": [ 2024-09-15T20:38:43.587 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.587 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:38:43.587 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:37:12.291089+0000", 2024-09-15T20:38:43.587 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:38:43.587 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:38:43.587 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:38:43.587 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:38:43.587 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:34:55.253095+0000", 2024-09-15T20:38:43.587 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:38:43.587 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.587 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.587 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:38:43.587 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:37:12.865088+0000", 2024-09-15T20:38:43.587 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:38:43.588 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:38:43.588 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:38:43.588 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:38:43.588 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:35:31.139386+0000", 2024-09-15T20:38:43.588 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:38:43.588 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.588 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.588 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:38:43.588 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:37:13.435960+0000", 2024-09-15T20:38:43.588 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:38:43.588 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:38:43.588 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:38:43.589 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:38:43.589 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:36:07.391280+0000", 2024-09-15T20:38:43.589 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:38:43.589 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.589 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:43.589 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap": [], 2024-09-15T20:38:43.589 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap_items": [], 2024-09-15T20:38:43.589 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap_primaries": [], 2024-09-15T20:38:43.589 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_temp": [], 2024-09-15T20:38:43.589 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_temp": [], 2024-09-15T20:38:43.589 INFO:tasks.workunit.client.0.smithi139.stdout: "blocklist": {}, 2024-09-15T20:38:43.589 INFO:tasks.workunit.client.0.smithi139.stdout: "range_blocklist": {}, 2024-09-15T20:38:43.589 INFO:tasks.workunit.client.0.smithi139.stdout: "erasure_code_profiles": { 2024-09-15T20:38:43.589 INFO:tasks.workunit.client.0.smithi139.stdout: "default": { 2024-09-15T20:38:43.589 INFO:tasks.workunit.client.0.smithi139.stdout: "k": "2", 2024-09-15T20:38:43.590 INFO:tasks.workunit.client.0.smithi139.stdout: "m": "2", 2024-09-15T20:38:43.590 INFO:tasks.workunit.client.0.smithi139.stdout: "plugin": "jerasure", 2024-09-15T20:38:43.590 INFO:tasks.workunit.client.0.smithi139.stdout: "technique": "reed_sol_van" 2024-09-15T20:38:43.590 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.590 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.590 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_snaps_queue": [], 2024-09-15T20:38:43.590 INFO:tasks.workunit.client.0.smithi139.stdout: "new_removed_snaps": [], 2024-09-15T20:38:43.590 INFO:tasks.workunit.client.0.smithi139.stdout: "new_purged_snaps": [], 2024-09-15T20:38:43.590 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_node_flags": {}, 2024-09-15T20:38:43.590 INFO:tasks.workunit.client.0.smithi139.stdout: "device_class_flags": {}, 2024-09-15T20:38:43.590 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode": { 2024-09-15T20:38:43.590 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode_enabled": false, 2024-09-15T20:38:43.590 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_bucket_count": 0, 2024-09-15T20:38:43.590 INFO:tasks.workunit.client.0.smithi139.stdout: "degraded_stretch_mode": 0, 2024-09-15T20:38:43.591 INFO:tasks.workunit.client.0.smithi139.stdout: "recovering_stretch_mode": 0, 2024-09-15T20:38:43.591 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode_bucket": 0 2024-09-15T20:38:43.591 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.591 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.591 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_metadata": [ 2024-09-15T20:38:43.591 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.591 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:38:43.591 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:38:43.591 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6804/3684664553,v1:127.0.0.1:6805/3684664553]", 2024-09-15T20:38:43.591 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:38:43.591 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:38:43.591 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:38:43.591 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:38:43.591 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:38:43.592 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:38:43.592 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:38:43.592 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:38:43.592 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:38:43.592 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:38:43.592 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:38:43.592 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-15T20:38:43.592 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:38:43.592 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:38:43.592 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:38:43.592 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:38:43.592 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:38:43.592 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:38:43.592 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:38:43.593 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:38:43.593 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:38:43.593 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:38:43.593 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:34:28.822135Z", 2024-09-15T20:38:43.593 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:38:43.593 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:38:43.593 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:38:43.593 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:38:43.593 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:38:43.593 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:38:43.593 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:38:43.593 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6802/3684664553,v1:127.0.0.1:6803/3684664553]", 2024-09-15T20:38:43.593 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:38:43.594 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3684664553,v1:127.0.0.1:6809/3684664553]", 2024-09-15T20:38:43.594 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3684664553,v1:127.0.0.1:6807/3684664553]", 2024-09-15T20:38:43.594 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:38:43.594 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:38:43.594 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:38:43.594 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:38:43.594 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:38:43.594 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:38:43.594 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:38:43.594 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:38:43.594 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:38:43.594 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/0", 2024-09-15T20:38:43.594 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:38:43.595 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:38:43.595 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:38:43.595 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.595 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.595 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:38:43.595 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:38:43.595 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6812/351107530,v1:127.0.0.1:6813/351107530]", 2024-09-15T20:38:43.595 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:38:43.595 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:38:43.595 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:38:43.595 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:38:43.595 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:38:43.596 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:38:43.596 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:38:43.596 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:38:43.596 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:38:43.596 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:38:43.596 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:38:43.596 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-15T20:38:43.596 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:38:43.596 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:38:43.596 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:38:43.596 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:38:43.596 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:38:43.596 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:38:43.597 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:38:43.597 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:38:43.597 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:38:43.597 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:38:43.597 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:35:03.011496Z", 2024-09-15T20:38:43.597 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:38:43.597 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:38:43.597 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:38:43.597 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:38:43.597 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:38:43.597 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:38:43.597 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:38:43.597 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6810/351107530,v1:127.0.0.1:6811/351107530]", 2024-09-15T20:38:43.597 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:38:43.598 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/351107530,v1:127.0.0.1:6817/351107530]", 2024-09-15T20:38:43.598 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/351107530,v1:127.0.0.1:6815/351107530]", 2024-09-15T20:38:43.598 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:38:43.598 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:38:43.598 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:38:43.598 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:38:43.598 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:38:43.598 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:38:43.599 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:38:43.599 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:38:43.599 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:38:43.599 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/1", 2024-09-15T20:38:43.599 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:38:43.599 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:38:43.599 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:38:43.599 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.599 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.599 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:38:43.599 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:38:43.599 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6812/3676612316,v1:127.0.0.1:6813/3676612316]", 2024-09-15T20:38:43.599 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:38:43.599 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:38:43.600 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:38:43.600 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:38:43.600 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:38:43.600 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:38:43.600 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:38:43.600 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:38:43.600 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:38:43.600 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:38:43.600 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:38:43.600 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-15T20:38:43.600 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:38:43.600 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:38:43.601 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:38:43.601 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:38:43.601 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:38:43.601 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:38:43.601 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:38:43.601 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:38:43.601 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:38:43.601 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:38:43.601 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:35:38.971291Z", 2024-09-15T20:38:43.601 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:38:43.601 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:38:43.601 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:38:43.601 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:38:43.601 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:38:43.602 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:38:43.602 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:38:43.602 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6810/3676612316,v1:127.0.0.1:6811/3676612316]", 2024-09-15T20:38:43.602 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:38:43.602 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3676612316,v1:127.0.0.1:6817/3676612316]", 2024-09-15T20:38:43.603 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3676612316,v1:127.0.0.1:6815/3676612316]", 2024-09-15T20:38:43.603 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:38:43.603 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:38:43.603 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:38:43.603 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:38:43.603 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:38:43.603 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:38:43.603 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:38:43.603 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:38:43.603 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:38:43.604 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/2", 2024-09-15T20:38:43.604 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:38:43.604 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:38:43.604 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:38:43.604 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.604 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:43.604 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_clean_epochs": { 2024-09-15T20:38:43.604 INFO:tasks.workunit.client.0.smithi139.stdout: "min_last_epoch_clean": 0, 2024-09-15T20:38:43.604 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_clean": { 2024-09-15T20:38:43.604 INFO:tasks.workunit.client.0.smithi139.stdout: "per_pool": [ 2024-09-15T20:38:43.604 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.604 INFO:tasks.workunit.client.0.smithi139.stdout: "poolid": 1, 2024-09-15T20:38:43.604 INFO:tasks.workunit.client.0.smithi139.stdout: "floor": 36 2024-09-15T20:38:43.604 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.605 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.605 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.605 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_epochs": [ 2024-09-15T20:38:43.605 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.605 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:38:43.605 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 76 2024-09-15T20:38:43.605 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.605 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.605 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:38:43.605 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 80 2024-09-15T20:38:43.605 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.605 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.605 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.606 INFO:tasks.workunit.client.0.smithi139.stderr:report 1958623987 2024-09-15T20:38:43.606 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_first_committed": 1, 2024-09-15T20:38:43.606 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_last_committed": 87, 2024-09-15T20:38:43.606 INFO:tasks.workunit.client.0.smithi139.stdout: "crushmap": { 2024-09-15T20:38:43.606 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": [ 2024-09-15T20:38:43.606 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.607 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:38:43.607 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.0", 2024-09-15T20:38:43.607 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:38:43.607 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.607 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.607 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:38:43.607 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.1", 2024-09-15T20:38:43.607 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:38:43.607 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.607 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.607 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:38:43.607 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.2", 2024-09-15T20:38:43.607 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:38:43.607 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.608 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:43.608 INFO:tasks.workunit.client.0.smithi139.stdout: "types": [ 2024-09-15T20:38:43.608 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.608 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 0, 2024-09-15T20:38:43.608 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd" 2024-09-15T20:38:43.608 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.608 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.608 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:38:43.608 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "host" 2024-09-15T20:38:43.608 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.608 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.608 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 2, 2024-09-15T20:38:43.608 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "chassis" 2024-09-15T20:38:43.608 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.609 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.609 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 3, 2024-09-15T20:38:43.609 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "rack" 2024-09-15T20:38:43.609 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.609 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.609 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 4, 2024-09-15T20:38:43.609 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "row" 2024-09-15T20:38:43.609 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.609 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.609 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 5, 2024-09-15T20:38:43.609 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "pdu" 2024-09-15T20:38:43.609 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.609 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.609 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 6, 2024-09-15T20:38:43.610 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "pod" 2024-09-15T20:38:43.610 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.610 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.610 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 7, 2024-09-15T20:38:43.610 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "room" 2024-09-15T20:38:43.610 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.610 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.610 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 8, 2024-09-15T20:38:43.610 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "datacenter" 2024-09-15T20:38:43.610 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.610 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.610 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 9, 2024-09-15T20:38:43.610 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "zone" 2024-09-15T20:38:43.610 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.611 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.611 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 10, 2024-09-15T20:38:43.611 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "region" 2024-09-15T20:38:43.611 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.611 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.611 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:38:43.611 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "root" 2024-09-15T20:38:43.611 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.611 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:43.611 INFO:tasks.workunit.client.0.smithi139.stdout: "buckets": [ 2024-09-15T20:38:43.611 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.611 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -1, 2024-09-15T20:38:43.611 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "default", 2024-09-15T20:38:43.611 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:38:43.612 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "root", 2024-09-15T20:38:43.612 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:38:43.612 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:38:43.612 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:38:43.612 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:38:43.612 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.612 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -3, 2024-09-15T20:38:43.612 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:38:43.612 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:38:43.612 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.612 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.612 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.612 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.612 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -2, 2024-09-15T20:38:43.613 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "default~hdd", 2024-09-15T20:38:43.613 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:38:43.613 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "root", 2024-09-15T20:38:43.613 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:38:43.613 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:38:43.613 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:38:43.613 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:38:43.613 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.613 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -4, 2024-09-15T20:38:43.613 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:38:43.613 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:38:43.613 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.613 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.613 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.614 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.614 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -3, 2024-09-15T20:38:43.614 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "smithi139", 2024-09-15T20:38:43.614 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:38:43.614 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "host", 2024-09-15T20:38:43.614 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:38:43.614 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:38:43.614 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:38:43.614 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:38:43.614 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.614 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:38:43.614 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:38:43.614 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:38:43.615 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.615 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.615 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:38:43.615 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:38:43.615 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 1 2024-09-15T20:38:43.615 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.615 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.615 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:38:43.615 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:38:43.615 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 2 2024-09-15T20:38:43.615 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.615 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.616 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.616 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.616 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -4, 2024-09-15T20:38:43.616 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "smithi139~hdd", 2024-09-15T20:38:43.616 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:38:43.616 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "host", 2024-09-15T20:38:43.616 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:38:43.616 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:38:43.616 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:38:43.616 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:38:43.616 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.617 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:38:43.617 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:38:43.617 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:38:43.617 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.617 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.617 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:38:43.617 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:38:43.617 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 1 2024-09-15T20:38:43.617 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.617 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.617 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:38:43.617 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:38:43.617 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 2 2024-09-15T20:38:43.617 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.618 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.618 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.618 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:43.618 INFO:tasks.workunit.client.0.smithi139.stdout: "rules": [ 2024-09-15T20:38:43.618 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.618 INFO:tasks.workunit.client.0.smithi139.stdout: "rule_id": 0, 2024-09-15T20:38:43.618 INFO:tasks.workunit.client.0.smithi139.stdout: "rule_name": "replicated_rule", 2024-09-15T20:38:43.618 INFO:tasks.workunit.client.0.smithi139.stdout: "type": 1, 2024-09-15T20:38:43.618 INFO:tasks.workunit.client.0.smithi139.stdout: "steps": [ 2024-09-15T20:38:43.618 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.618 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "take", 2024-09-15T20:38:43.618 INFO:tasks.workunit.client.0.smithi139.stdout: "item": -1, 2024-09-15T20:38:43.618 INFO:tasks.workunit.client.0.smithi139.stdout: "item_name": "default" 2024-09-15T20:38:43.618 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.618 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.619 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "choose_firstn", 2024-09-15T20:38:43.619 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 0, 2024-09-15T20:38:43.619 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "osd" 2024-09-15T20:38:43.619 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.619 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.619 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "emit" 2024-09-15T20:38:43.619 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.619 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:38:43.619 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.619 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:43.619 INFO:tasks.workunit.client.0.smithi139.stdout: "tunables": { 2024-09-15T20:38:43.619 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_local_tries": 0, 2024-09-15T20:38:43.619 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_local_fallback_tries": 0, 2024-09-15T20:38:43.619 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_total_tries": 50, 2024-09-15T20:38:43.619 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_descend_once": 1, 2024-09-15T20:38:43.620 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_vary_r": 1, 2024-09-15T20:38:43.620 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_stable": 1, 2024-09-15T20:38:43.620 INFO:tasks.workunit.client.0.smithi139.stdout: "msr_descents": 100, 2024-09-15T20:38:43.620 INFO:tasks.workunit.client.0.smithi139.stdout: "msr_collision_tries": 100, 2024-09-15T20:38:43.620 INFO:tasks.workunit.client.0.smithi139.stdout: "straw_calc_version": 1, 2024-09-15T20:38:43.620 INFO:tasks.workunit.client.0.smithi139.stdout: "allowed_bucket_algs": 54, 2024-09-15T20:38:43.620 INFO:tasks.workunit.client.0.smithi139.stdout: "profile": "jewel", 2024-09-15T20:38:43.620 INFO:tasks.workunit.client.0.smithi139.stdout: "optimal_tunables": 1, 2024-09-15T20:38:43.620 INFO:tasks.workunit.client.0.smithi139.stdout: "legacy_tunables": 0, 2024-09-15T20:38:43.620 INFO:tasks.workunit.client.0.smithi139.stdout: "minimum_required_version": "jewel", 2024-09-15T20:38:43.620 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables": 1, 2024-09-15T20:38:43.620 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables2": 1, 2024-09-15T20:38:43.620 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v2_rules": 0, 2024-09-15T20:38:43.620 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables3": 1, 2024-09-15T20:38:43.621 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v3_rules": 0, 2024-09-15T20:38:43.621 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v4_buckets": 1, 2024-09-15T20:38:43.621 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables5": 1, 2024-09-15T20:38:43.621 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v5_rules": 0, 2024-09-15T20:38:43.621 INFO:tasks.workunit.client.0.smithi139.stdout: "has_msr_rules": 0 2024-09-15T20:38:43.621 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.621 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_args": {} 2024-09-15T20:38:43.621 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.621 INFO:tasks.workunit.client.0.smithi139.stdout: "fsmap": { 2024-09-15T20:38:43.621 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 1, 2024-09-15T20:38:43.621 INFO:tasks.workunit.client.0.smithi139.stdout: "btime": "2024-09-15T20:34:22:211262+0000", 2024-09-15T20:38:43.621 INFO:tasks.workunit.client.0.smithi139.stdout: "default_fscid": -1, 2024-09-15T20:38:43.621 INFO:tasks.workunit.client.0.smithi139.stdout: "compat": { 2024-09-15T20:38:43.621 INFO:tasks.workunit.client.0.smithi139.stdout: "compat": {}, 2024-09-15T20:38:43.622 INFO:tasks.workunit.client.0.smithi139.stdout: "ro_compat": {}, 2024-09-15T20:38:43.622 INFO:tasks.workunit.client.0.smithi139.stdout: "incompat": { 2024-09-15T20:38:43.622 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_1": "base v0.20", 2024-09-15T20:38:43.622 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_2": "client writeable ranges", 2024-09-15T20:38:43.622 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_3": "default file layouts on dirs", 2024-09-15T20:38:43.622 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_4": "dir inode in separate object", 2024-09-15T20:38:43.622 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_5": "mds uses versioned encoding", 2024-09-15T20:38:43.622 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-15T20:38:43.622 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_8": "no anchor table", 2024-09-15T20:38:43.622 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_9": "file layout v2", 2024-09-15T20:38:43.622 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_10": "snaprealm v2", 2024-09-15T20:38:43.622 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_11": "minor log segments", 2024-09-15T20:38:43.622 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_12": "quiesce subvolumes" 2024-09-15T20:38:43.622 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.622 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.623 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_flags": { 2024-09-15T20:38:43.623 INFO:tasks.workunit.client.0.smithi139.stdout: "enable_multiple": true, 2024-09-15T20:38:43.623 INFO:tasks.workunit.client.0.smithi139.stdout: "ever_enabled_multiple": true 2024-09-15T20:38:43.623 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.623 INFO:tasks.workunit.client.0.smithi139.stdout: "standbys": [], 2024-09-15T20:38:43.623 INFO:tasks.workunit.client.0.smithi139.stdout: "filesystems": [] 2024-09-15T20:38:43.623 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.623 INFO:tasks.workunit.client.0.smithi139.stdout: "mdsmap_first_committed": 1, 2024-09-15T20:38:43.623 INFO:tasks.workunit.client.0.smithi139.stdout: "mdsmap_last_committed": 1, 2024-09-15T20:38:43.623 INFO:tasks.workunit.client.0.smithi139.stdout: "auth": { 2024-09-15T20:38:43.623 INFO:tasks.workunit.client.0.smithi139.stdout: "first_committed": 1, 2024-09-15T20:38:43.623 INFO:tasks.workunit.client.0.smithi139.stdout: "last_committed": 5, 2024-09-15T20:38:43.623 INFO:tasks.workunit.client.0.smithi139.stdout: "num_secrets": 3 2024-09-15T20:38:43.623 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.623 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg": 32, 2024-09-15T20:38:43.624 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_active": 32, 2024-09-15T20:38:43.624 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_unknown": 0, 2024-09-15T20:38:43.624 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osd": 3, 2024-09-15T20:38:43.624 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_sum": { 2024-09-15T20:38:43.624 INFO:tasks.workunit.client.0.smithi139.stdout: "stat_sum": { 2024-09-15T20:38:43.624 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes": 61183, 2024-09-15T20:38:43.624 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects": 100, 2024-09-15T20:38:43.624 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_clones": 0, 2024-09-15T20:38:43.624 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_copies": 300, 2024-09-15T20:38:43.624 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T20:38:43.624 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing": 0, 2024-09-15T20:38:43.624 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_degraded": 100, 2024-09-15T20:38:43.624 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_misplaced": 0, 2024-09-15T20:38:43.624 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_unfound": 0, 2024-09-15T20:38:43.625 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_dirty": 100, 2024-09-15T20:38:43.625 INFO:tasks.workunit.client.0.smithi139.stdout: "num_whiteouts": 0, 2024-09-15T20:38:43.625 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read": 0, 2024-09-15T20:38:43.625 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read_kb": 0, 2024-09-15T20:38:43.625 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write": 59, 2024-09-15T20:38:43.625 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write_kb": 59, 2024-09-15T20:38:43.625 INFO:tasks.workunit.client.0.smithi139.stdout: "num_scrub_errors": 0, 2024-09-15T20:38:43.625 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T20:38:43.625 INFO:tasks.workunit.client.0.smithi139.stdout: "num_deep_scrub_errors": 0, 2024-09-15T20:38:43.625 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_recovered": 0, 2024-09-15T20:38:43.625 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_recovered": 0, 2024-09-15T20:38:43.625 INFO:tasks.workunit.client.0.smithi139.stdout: "num_keys_recovered": 0, 2024-09-15T20:38:43.625 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_omap": 0, 2024-09-15T20:38:43.626 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T20:38:43.626 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T20:38:43.626 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush": 0, 2024-09-15T20:38:43.626 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_kb": 0, 2024-09-15T20:38:43.626 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict": 0, 2024-09-15T20:38:43.626 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_kb": 0, 2024-09-15T20:38:43.626 INFO:tasks.workunit.client.0.smithi139.stdout: "num_promote": 0, 2024-09-15T20:38:43.626 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_high": 0, 2024-09-15T20:38:43.626 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_low": 0, 2024-09-15T20:38:43.626 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_some": 0, 2024-09-15T20:38:43.626 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_full": 0, 2024-09-15T20:38:43.626 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_pinned": 0, 2024-09-15T20:38:43.626 INFO:tasks.workunit.client.0.smithi139.stdout: "num_legacy_snapsets": 0, 2024-09-15T20:38:43.627 INFO:tasks.workunit.client.0.smithi139.stdout: "num_large_omap_objects": 0, 2024-09-15T20:38:43.627 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_manifest": 0, 2024-09-15T20:38:43.627 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_bytes": 0, 2024-09-15T20:38:43.627 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_keys": 0, 2024-09-15T20:38:43.627 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_repaired": 0 2024-09-15T20:38:43.627 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.627 INFO:tasks.workunit.client.0.smithi139.stdout: "store_stats": { 2024-09-15T20:38:43.627 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 0, 2024-09-15T20:38:43.627 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 0, 2024-09-15T20:38:43.627 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:38:43.627 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 0, 2024-09-15T20:38:43.627 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 0, 2024-09-15T20:38:43.627 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:38:43.627 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:38:43.628 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:38:43.628 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 0, 2024-09-15T20:38:43.628 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 0 2024-09-15T20:38:43.628 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.628 INFO:tasks.workunit.client.0.smithi139.stdout: "log_size": 554, 2024-09-15T20:38:43.628 INFO:tasks.workunit.client.0.smithi139.stdout: "ondisk_log_size": 554, 2024-09-15T20:38:43.628 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 64, 2024-09-15T20:38:43.628 INFO:tasks.workunit.client.0.smithi139.stdout: "acting": 64, 2024-09-15T20:38:43.628 INFO:tasks.workunit.client.0.smithi139.stdout: "num_store_stats": 0 2024-09-15T20:38:43.628 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.628 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_sum": { 2024-09-15T20:38:43.628 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 0, 2024-09-15T20:38:43.628 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 0, 2024-09-15T20:38:43.628 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pgs": 96, 2024-09-15T20:38:43.629 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osds": 3, 2024-09-15T20:38:43.629 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_osds": 3, 2024-09-15T20:38:43.629 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T20:38:43.629 INFO:tasks.workunit.client.0.smithi139.stdout: "kb": 314572800, 2024-09-15T20:38:43.629 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used": 84640, 2024-09-15T20:38:43.629 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_data": 3088, 2024-09-15T20:38:43.629 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_omap": 10, 2024-09-15T20:38:43.629 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_meta": 81461, 2024-09-15T20:38:43.629 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_avail": 314488160, 2024-09-15T20:38:43.629 INFO:tasks.workunit.client.0.smithi139.stdout: "statfs": { 2024-09-15T20:38:43.629 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 322122547200, 2024-09-15T20:38:43.629 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 322035875840, 2024-09-15T20:38:43.629 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:38:43.630 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 3162112, 2024-09-15T20:38:43.630 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 824354, 2024-09-15T20:38:43.630 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:38:43.630 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:38:43.630 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:38:43.630 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 10399, 2024-09-15T20:38:43.630 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 83416929 2024-09-15T20:38:43.630 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.630 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_peers": [], 2024-09-15T20:38:43.630 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_trim_queue_len": 0, 2024-09-15T20:38:43.630 INFO:tasks.workunit.client.0.smithi139.stdout: "num_snap_trimming": 0, 2024-09-15T20:38:43.630 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shards_repaired": 0, 2024-09-15T20:38:43.630 INFO:tasks.workunit.client.0.smithi139.stdout: "op_queue_age_hist": { 2024-09-15T20:38:43.630 INFO:tasks.workunit.client.0.smithi139.stdout: "histogram": [], 2024-09-15T20:38:43.631 INFO:tasks.workunit.client.0.smithi139.stdout: "upper_bound": 1 2024-09-15T20:38:43.631 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.631 INFO:tasks.workunit.client.0.smithi139.stdout: "perf_stat": { 2024-09-15T20:38:43.631 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ms": 100, 2024-09-15T20:38:43.631 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ms": 100, 2024-09-15T20:38:43.631 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ns": 100000000, 2024-09-15T20:38:43.631 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ns": 100000000 2024-09-15T20:38:43.631 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.631 INFO:tasks.workunit.client.0.smithi139.stdout: "alerts": [], 2024-09-15T20:38:43.631 INFO:tasks.workunit.client.0.smithi139.stdout: "network_ping_times": [] 2024-09-15T20:38:43.631 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.631 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_sum_by_class": { 2024-09-15T20:38:43.631 INFO:tasks.workunit.client.0.smithi139.stdout: "hdd": { 2024-09-15T20:38:43.631 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 0, 2024-09-15T20:38:43.632 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 0, 2024-09-15T20:38:43.632 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pgs": 96, 2024-09-15T20:38:43.632 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osds": 3, 2024-09-15T20:38:43.632 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_osds": 3, 2024-09-15T20:38:43.632 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T20:38:43.632 INFO:tasks.workunit.client.0.smithi139.stdout: "kb": 314572800, 2024-09-15T20:38:43.632 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used": 84640, 2024-09-15T20:38:43.632 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_data": 3088, 2024-09-15T20:38:43.632 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_omap": 10, 2024-09-15T20:38:43.632 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_meta": 81461, 2024-09-15T20:38:43.632 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_avail": 314488160, 2024-09-15T20:38:43.632 INFO:tasks.workunit.client.0.smithi139.stdout: "statfs": { 2024-09-15T20:38:43.632 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 322122547200, 2024-09-15T20:38:43.633 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 322035875840, 2024-09-15T20:38:43.633 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:38:43.633 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 3162112, 2024-09-15T20:38:43.633 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 824354, 2024-09-15T20:38:43.633 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:38:43.633 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:38:43.633 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:38:43.633 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 10399, 2024-09-15T20:38:43.633 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 83416929 2024-09-15T20:38:43.633 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.633 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_peers": [], 2024-09-15T20:38:43.633 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_trim_queue_len": 0, 2024-09-15T20:38:43.633 INFO:tasks.workunit.client.0.smithi139.stdout: "num_snap_trimming": 0, 2024-09-15T20:38:43.634 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shards_repaired": 0, 2024-09-15T20:38:43.634 INFO:tasks.workunit.client.0.smithi139.stdout: "op_queue_age_hist": { 2024-09-15T20:38:43.634 INFO:tasks.workunit.client.0.smithi139.stdout: "histogram": [], 2024-09-15T20:38:43.634 INFO:tasks.workunit.client.0.smithi139.stdout: "upper_bound": 1 2024-09-15T20:38:43.634 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.634 INFO:tasks.workunit.client.0.smithi139.stdout: "perf_stat": { 2024-09-15T20:38:43.634 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ms": 100, 2024-09-15T20:38:43.634 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ms": 100, 2024-09-15T20:38:43.634 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ns": 100000000, 2024-09-15T20:38:43.634 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ns": 100000000 2024-09-15T20:38:43.634 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.634 INFO:tasks.workunit.client.0.smithi139.stdout: "alerts": [], 2024-09-15T20:38:43.634 INFO:tasks.workunit.client.0.smithi139.stdout: "network_ping_times": [] 2024-09-15T20:38:43.635 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.635 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.635 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_stats": [ 2024-09-15T20:38:43.635 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.635 INFO:tasks.workunit.client.0.smithi139.stdout: "poolid": 1, 2024-09-15T20:38:43.635 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg": 32, 2024-09-15T20:38:43.635 INFO:tasks.workunit.client.0.smithi139.stdout: "stat_sum": { 2024-09-15T20:38:43.635 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes": 61183, 2024-09-15T20:38:43.635 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects": 100, 2024-09-15T20:38:43.635 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_clones": 0, 2024-09-15T20:38:43.635 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_copies": 300, 2024-09-15T20:38:43.635 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T20:38:43.635 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing": 0, 2024-09-15T20:38:43.635 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_degraded": 100, 2024-09-15T20:38:43.636 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_misplaced": 0, 2024-09-15T20:38:43.636 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_unfound": 0, 2024-09-15T20:38:43.636 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_dirty": 100, 2024-09-15T20:38:43.636 INFO:tasks.workunit.client.0.smithi139.stdout: "num_whiteouts": 0, 2024-09-15T20:38:43.636 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read": 0, 2024-09-15T20:38:43.636 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read_kb": 0, 2024-09-15T20:38:43.636 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write": 59, 2024-09-15T20:38:43.636 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write_kb": 59, 2024-09-15T20:38:43.636 INFO:tasks.workunit.client.0.smithi139.stdout: "num_scrub_errors": 0, 2024-09-15T20:38:43.636 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T20:38:43.636 INFO:tasks.workunit.client.0.smithi139.stdout: "num_deep_scrub_errors": 0, 2024-09-15T20:38:43.636 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_recovered": 0, 2024-09-15T20:38:43.637 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_recovered": 0, 2024-09-15T20:38:43.637 INFO:tasks.workunit.client.0.smithi139.stdout: "num_keys_recovered": 0, 2024-09-15T20:38:43.637 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_omap": 0, 2024-09-15T20:38:43.637 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T20:38:43.637 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T20:38:43.637 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush": 0, 2024-09-15T20:38:43.637 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_kb": 0, 2024-09-15T20:38:43.637 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict": 0, 2024-09-15T20:38:43.637 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_kb": 0, 2024-09-15T20:38:43.637 INFO:tasks.workunit.client.0.smithi139.stdout: "num_promote": 0, 2024-09-15T20:38:43.637 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_high": 0, 2024-09-15T20:38:43.637 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_low": 0, 2024-09-15T20:38:43.637 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_some": 0, 2024-09-15T20:38:43.638 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_full": 0, 2024-09-15T20:38:43.638 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_pinned": 0, 2024-09-15T20:38:43.638 INFO:tasks.workunit.client.0.smithi139.stdout: "num_legacy_snapsets": 0, 2024-09-15T20:38:43.638 INFO:tasks.workunit.client.0.smithi139.stdout: "num_large_omap_objects": 0, 2024-09-15T20:38:43.638 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_manifest": 0, 2024-09-15T20:38:43.638 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_bytes": 0, 2024-09-15T20:38:43.638 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_keys": 0, 2024-09-15T20:38:43.638 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_repaired": 0 2024-09-15T20:38:43.638 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.638 INFO:tasks.workunit.client.0.smithi139.stdout: "store_stats": { 2024-09-15T20:38:43.638 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 0, 2024-09-15T20:38:43.638 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 0, 2024-09-15T20:38:43.638 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:38:43.639 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 1220608, 2024-09-15T20:38:43.639 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 185266, 2024-09-15T20:38:43.639 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:38:43.639 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:38:43.639 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:38:43.639 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 0, 2024-09-15T20:38:43.639 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 0 2024-09-15T20:38:43.639 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.639 INFO:tasks.workunit.client.0.smithi139.stdout: "log_size": 554, 2024-09-15T20:38:43.639 INFO:tasks.workunit.client.0.smithi139.stdout: "ondisk_log_size": 554, 2024-09-15T20:38:43.639 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 64, 2024-09-15T20:38:43.639 INFO:tasks.workunit.client.0.smithi139.stdout: "acting": 64, 2024-09-15T20:38:43.639 INFO:tasks.workunit.client.0.smithi139.stdout: "num_store_stats": 3 2024-09-15T20:38:43.639 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.640 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:43.640 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_stats": [ 2024-09-15T20:38:43.640 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.640 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:38:43.640 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 326417514517 2024-09-15T20:38:43.640 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.640 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.640 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:38:43.640 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 42949672992 2024-09-15T20:38:43.640 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.640 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.640 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:38:43.640 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 343597383700 2024-09-15T20:38:43.641 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.641 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:43.641 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_by_state": [ 2024-09-15T20:38:43.641 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.641 INFO:tasks.workunit.client.0.smithi139.stdout: "state": "active+undersized+degraded", 2024-09-15T20:38:43.641 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 32 2024-09-15T20:38:43.641 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.641 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:43.641 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_by_osd": [ 2024-09-15T20:38:43.641 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.641 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:38:43.641 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 16, 2024-09-15T20:38:43.641 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 32, 2024-09-15T20:38:43.642 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:38:43.642 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.642 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.642 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:38:43.642 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 16, 2024-09-15T20:38:43.642 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 32, 2024-09-15T20:38:43.642 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:38:43.642 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.642 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.642 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:38:43.642 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 0, 2024-09-15T20:38:43.642 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 0, 2024-09-15T20:38:43.642 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:38:43.642 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.643 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:43.643 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snaps": [ 2024-09-15T20:38:43.643 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:38:43.643 INFO:tasks.workunit.client.0.smithi139.stdout: "pool": 1, 2024-09-15T20:38:43.643 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snaps": {} 2024-09-15T20:38:43.643 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.643 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:38:43.643 INFO:tasks.workunit.client.0.smithi139.stdout: "servicemap": { 2024-09-15T20:38:43.643 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 7, 2024-09-15T20:38:43.643 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:38:27.913429+0000", 2024-09-15T20:38:43.643 INFO:tasks.workunit.client.0.smithi139.stdout: "services": {} 2024-09-15T20:38:43.643 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:38:43.643 INFO:tasks.workunit.client.0.smithi139.stdout: "mgrstat_first_committed": 1, 2024-09-15T20:38:43.643 INFO:tasks.workunit.client.0.smithi139.stdout: "mgrstat_last_committed": 119, 2024-09-15T20:38:43.643 INFO:tasks.workunit.client.0.smithi139.stdout: "logm_first_committed": 1, 2024-09-15T20:38:43.644 INFO:tasks.workunit.client.0.smithi139.stdout: "logm_last_committed": 539, 2024-09-15T20:38:43.644 INFO:tasks.workunit.client.0.smithi139.stdout: "paxos": { 2024-09-15T20:38:43.644 INFO:tasks.workunit.client.0.smithi139.stdout: "first_committed": 1, 2024-09-15T20:38:43.644 INFO:tasks.workunit.client.0.smithi139.stdout: "last_committed": 731, 2024-09-15T20:38:43.644 INFO:tasks.workunit.client.0.smithi139.stdout: "last_pn": 0, 2024-09-15T20:38:43.644 INFO:tasks.workunit.client.0.smithi139.stdout: "accepted_pn": 0 2024-09-15T20:38:43.644 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:38:43.644 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:38:43.644 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-15T20:38:43.645 INFO:tasks.workunit.client.0.smithi139.stdout:delay recovery 2024-09-15T20:38:43.645 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: echo 'delay recovery' 2024-09-15T20:38:43.645 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph pg dump pgs 2024-09-15T20:38:43.952 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:38:43.952 INFO:tasks.workunit.client.0.smithi139.stdout:1.1f 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394529+0000 41'29 87:103 [2,0] 2 [2,0] 2 37'28 2024-09-15T20:36:40.657710+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:26:32.119186+0000 2 0 2024-09-15T20:38:43.952 INFO:tasks.workunit.client.0.smithi139.stdout:1.1e 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394381+0000 46'29 87:116 [2,0] 2 [2,0] 2 40'28 2024-09-15T20:36:45.584734+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:06:07.931182+0000 2 0 2024-09-15T20:38:43.952 INFO:tasks.workunit.client.0.smithi139.stdout:1.1d 4 0 4 0 0 2468 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:29.424550+0000 42'31 87:114 [2,0] 2 [2,0] 2 36'29 2024-09-15T20:36:37.493943+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:34:28.615082+0000 2 0 2024-09-15T20:38:43.952 INFO:tasks.workunit.client.0.smithi139.stdout:1.b 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.396794+0000 45'29 87:115 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:44.549410+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:27:32.595447+0000 2 0 2024-09-15T20:38:43.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.a 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.368887+0000 39'29 87:93 [2,0] 2 [2,0] 2 39'29 2024-09-15T20:36:56.804751+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:38:28.674672+0000 0 0 2024-09-15T20:38:43.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.9 5 0 5 0 0 3085 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.368691+0000 45'29 87:118 [2,0] 2 [2,0] 2 45'29 2024-09-15T20:36:51.656322+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:14:38.500654+0000 3 0 2024-09-15T20:38:43.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.8 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.499768+0000 45'30 87:106 [2,0] 2 [2,0] 2 45'30 2024-09-15T20:36:50.633607+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:43.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.7 2 0 2 0 0 1138 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:29.424254+0000 38'28 87:80 [2,0] 2 [2,0] 2 38'28 2024-09-15T20:36:48.576911+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:43.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.6 4 0 4 0 0 2468 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.410719+0000 42'30 87:102 [0,2] 0 [0,2] 0 42'30 2024-09-15T20:36:46.625811+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:02:22.839337+0000 2 0 2024-09-15T20:38:43.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.5 6 0 6 0 0 3702 0 0 20 12 20 active+undersized+degraded 2024-09-15T20:37:28.369909+0000 45'32 87:127 [2,0] 2 [2,0] 2 40'30 2024-09-15T20:36:40.468115+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:22:28.237620+0000 4 0 2024-09-15T20:38:43.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.4 2 0 2 0 0 1234 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.399006+0000 40'28 87:104 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:38.561548+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:24:41.193931+0000 1 0 2024-09-15T20:38:43.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.3 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.046599+0000 47'30 87:108 [2,0] 2 [2,0] 2 47'30 2024-09-15T20:36:54.739486+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:43.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.2 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.409088+0000 42'28 87:99 [0,2] 0 [0,2] 0 42'28 2024-09-15T20:36:44.289898+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:59:17.740071+0000 1 0 2024-09-15T20:38:43.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.c 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.395833+0000 44'29 87:116 [0,2] 0 [0,2] 0 38'28 2024-09-15T20:36:43.561129+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:59:06.361170+0000 2 0 2024-09-15T20:38:43.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.1 2 0 2 0 0 1234 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:29.499685+0000 44'29 87:104 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:42.549754+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:03:26.939083+0000 0 0 2024-09-15T20:38:43.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.1c 2 0 2 0 0 1234 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.398522+0000 36'29 87:95 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:34.684895+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:11:58.732534+0000 1 0 2024-09-15T20:38:43.954 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.409170+0000 45'29 87:119 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:31.575682+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:19:06.387325+0000 1 0 2024-09-15T20:38:43.954 INFO:tasks.workunit.client.0.smithi139.stdout:1.d 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394417+0000 45'29 87:113 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:39.548519+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:37:36.633002+0000 1 0 2024-09-15T20:38:43.954 INFO:tasks.workunit.client.0.smithi139.stdout:1.e 3 0 3 0 0 2105 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.097547+0000 87'30 87:94 [2,0] 2 [2,0] 2 44'29 2024-09-15T20:36:49.578000+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:38:30.733965+0000 0 0 2024-09-15T20:38:43.954 INFO:tasks.workunit.client.0.smithi139.stdout:1.f 4 0 4 0 0 2468 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.396239+0000 41'29 87:103 [2,0] 2 [2,0] 2 34'28 2024-09-15T20:36:32.588850+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:09:31.037907+0000 3 0 2024-09-15T20:38:43.954 INFO:tasks.workunit.client.0.smithi139.stdout:1.10 3 0 3 0 0 1851 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.397764+0000 45'30 87:120 [0,2] 0 [0,2] 0 38'29 2024-09-15T20:36:39.578100+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:37:31.289109+0000 2 0 2024-09-15T20:38:43.954 INFO:tasks.workunit.client.0.smithi139.stdout:1.11 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.398513+0000 37'29 87:90 [0,2] 0 [0,2] 0 36'28 2024-09-15T20:36:36.444590+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:48:59.386094+0000 2 0 2024-09-15T20:38:43.954 INFO:tasks.workunit.client.0.smithi139.stdout:1.12 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.397545+0000 36'28 87:105 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:35.623615+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:48:18.440517+0000 0 0 2024-09-15T20:38:43.954 INFO:tasks.workunit.client.0.smithi139.stdout:1.13 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.396874+0000 45'30 87:108 [0,2] 0 [0,2] 0 45'30 2024-09-15T20:36:55.766014+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:38:28.994911+0000 0 0 2024-09-15T20:38:43.954 INFO:tasks.workunit.client.0.smithi139.stdout:1.14 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:29.122393+0000 45'29 87:114 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:37.554032+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:59:56.311369+0000 1 0 2024-09-15T20:38:43.954 INFO:tasks.workunit.client.0.smithi139.stdout:1.15 3 0 3 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.411771+0000 41'29 87:105 [0,2] 0 [0,2] 0 41'29 2024-09-15T20:36:41.506894+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:43.954 INFO:tasks.workunit.client.0.smithi139.stdout:1.16 4 0 4 0 0 2468 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:28.410181+0000 41'31 87:115 [0,2] 0 [0,2] 0 41'31 2024-09-15T20:36:47.576518+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:29:37.611785+0000 3 0 2024-09-15T20:38:43.954 INFO:tasks.workunit.client.0.smithi139.stdout:1.17 1 0 1 0 0 520 0 0 15 12 15 active+undersized+degraded 2024-09-15T20:37:28.409266+0000 32'27 87:68 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:42.639678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:43.954 INFO:tasks.workunit.client.0.smithi139.stdout:1.18 3 0 3 0 0 1851 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.394810+0000 45'30 87:119 [2,0] 2 [2,0] 2 34'28 2024-09-15T20:36:36.622678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:29:24.808950+0000 1 0 2024-09-15T20:38:43.955 INFO:tasks.workunit.client.0.smithi139.stdout:1.19 2 0 2 0 0 1137 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.398489+0000 34'28 87:81 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:52.759747+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:43.955 INFO:tasks.workunit.client.0.smithi139.stdout:1.1a 4 0 4 0 0 2468 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.369086+0000 44'30 87:116 [2,0] 2 [2,0] 2 44'30 2024-09-15T20:36:53.716793+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:57:48.429965+0000 3 0 2024-09-15T20:38:43.955 INFO:tasks.workunit.client.0.smithi139.stdout:1.1b 5 0 5 0 0 3085 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:28.396910+0000 45'31 87:124 [0,2] 0 [0,2] 0 41'30 2024-09-15T20:36:41.658323+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:57:15.784191+0000 4 0 2024-09-15T20:38:43.955 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:38:43.955 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:38:43.955 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:38:43.967 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-09-15T20:38:43.969 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.0 2024-09-15T20:38:43.969 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T20:38:43.969 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T20:38:43.969 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:38:43.969 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:38:43.969 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:38:43.970 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.0.asok 2024-09-15T20:38:43.970 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-09-15T20:38:43.970 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.0.asok set_recovery_delay 100000 2024-09-15T20:38:44.075 INFO:tasks.workunit.client.0.smithi139.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T20:38:44.086 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-09-15T20:38:44.087 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.2 2024-09-15T20:38:44.087 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-15T20:38:44.088 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-15T20:38:44.088 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:38:44.088 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:38:44.088 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:38:44.089 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.2.asok 2024-09-15T20:38:44.089 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-09-15T20:38:44.089 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.2.asok set_recovery_delay 100000 2024-09-15T20:38:44.208 INFO:tasks.workunit.client.0.smithi139.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T20:38:44.221 INFO:tasks.workunit.client.0.smithi139.stdout:revive divergent 1 2024-09-15T20:38:44.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:769: TEST_divergent_3: echo 'revive divergent 1' 2024-09-15T20:38:44.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2024-09-15T20:38:44.626 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:38:44.626 INFO:tasks.workunit.client.0.smithi139.stdout:1.1f 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394529+0000 41'29 87:103 [2,0] 2 [2,0] 2 37'28 2024-09-15T20:36:40.657710+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:26:32.119186+0000 2 0 2024-09-15T20:38:44.626 INFO:tasks.workunit.client.0.smithi139.stdout:1.1e 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394381+0000 46'29 87:116 [2,0] 2 [2,0] 2 40'28 2024-09-15T20:36:45.584734+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:06:07.931182+0000 2 0 2024-09-15T20:38:44.627 INFO:tasks.workunit.client.0.smithi139.stdout:1.1d 4 0 4 0 0 2468 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:29.424550+0000 42'31 87:114 [2,0] 2 [2,0] 2 36'29 2024-09-15T20:36:37.493943+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:34:28.615082+0000 2 0 2024-09-15T20:38:44.627 INFO:tasks.workunit.client.0.smithi139.stdout:1.b 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.396794+0000 45'29 87:115 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:44.549410+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:27:32.595447+0000 2 0 2024-09-15T20:38:44.627 INFO:tasks.workunit.client.0.smithi139.stdout:1.a 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.368887+0000 39'29 87:93 [2,0] 2 [2,0] 2 39'29 2024-09-15T20:36:56.804751+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:38:28.674672+0000 0 0 2024-09-15T20:38:44.627 INFO:tasks.workunit.client.0.smithi139.stdout:1.9 5 0 5 0 0 3085 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.368691+0000 45'29 87:118 [2,0] 2 [2,0] 2 45'29 2024-09-15T20:36:51.656322+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:14:38.500654+0000 3 0 2024-09-15T20:38:44.627 INFO:tasks.workunit.client.0.smithi139.stdout:1.8 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.499768+0000 45'30 87:106 [2,0] 2 [2,0] 2 45'30 2024-09-15T20:36:50.633607+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:44.627 INFO:tasks.workunit.client.0.smithi139.stdout:1.7 2 0 2 0 0 1138 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:29.424254+0000 38'28 87:80 [2,0] 2 [2,0] 2 38'28 2024-09-15T20:36:48.576911+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:44.627 INFO:tasks.workunit.client.0.smithi139.stdout:1.6 4 0 4 0 0 2468 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.410719+0000 42'30 87:102 [0,2] 0 [0,2] 0 42'30 2024-09-15T20:36:46.625811+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:02:22.839337+0000 2 0 2024-09-15T20:38:44.627 INFO:tasks.workunit.client.0.smithi139.stdout:1.5 6 0 6 0 0 3702 0 0 20 12 20 active+undersized+degraded 2024-09-15T20:37:28.369909+0000 45'32 87:127 [2,0] 2 [2,0] 2 40'30 2024-09-15T20:36:40.468115+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:22:28.237620+0000 4 0 2024-09-15T20:38:44.627 INFO:tasks.workunit.client.0.smithi139.stdout:1.4 2 0 2 0 0 1234 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.399006+0000 40'28 87:104 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:38.561548+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:24:41.193931+0000 1 0 2024-09-15T20:38:44.627 INFO:tasks.workunit.client.0.smithi139.stdout:1.3 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.046599+0000 47'30 87:108 [2,0] 2 [2,0] 2 47'30 2024-09-15T20:36:54.739486+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:44.627 INFO:tasks.workunit.client.0.smithi139.stdout:1.2 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.409088+0000 42'28 87:99 [0,2] 0 [0,2] 0 42'28 2024-09-15T20:36:44.289898+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:59:17.740071+0000 1 0 2024-09-15T20:38:44.627 INFO:tasks.workunit.client.0.smithi139.stdout:1.c 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.395833+0000 44'29 87:116 [0,2] 0 [0,2] 0 38'28 2024-09-15T20:36:43.561129+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:59:06.361170+0000 2 0 2024-09-15T20:38:44.627 INFO:tasks.workunit.client.0.smithi139.stdout:1.1 2 0 2 0 0 1234 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:29.499685+0000 44'29 87:104 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:42.549754+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:03:26.939083+0000 0 0 2024-09-15T20:38:44.628 INFO:tasks.workunit.client.0.smithi139.stdout:1.1c 2 0 2 0 0 1234 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.398522+0000 36'29 87:95 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:34.684895+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:11:58.732534+0000 1 0 2024-09-15T20:38:44.628 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.409170+0000 45'29 87:119 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:31.575682+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:19:06.387325+0000 1 0 2024-09-15T20:38:44.628 INFO:tasks.workunit.client.0.smithi139.stdout:1.d 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394417+0000 45'29 87:113 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:39.548519+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:37:36.633002+0000 1 0 2024-09-15T20:38:44.628 INFO:tasks.workunit.client.0.smithi139.stdout:1.e 3 0 3 0 0 2105 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.097547+0000 87'30 87:94 [2,0] 2 [2,0] 2 44'29 2024-09-15T20:36:49.578000+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:38:30.733965+0000 0 0 2024-09-15T20:38:44.628 INFO:tasks.workunit.client.0.smithi139.stdout:1.f 4 0 4 0 0 2468 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.396239+0000 41'29 87:103 [2,0] 2 [2,0] 2 34'28 2024-09-15T20:36:32.588850+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:09:31.037907+0000 3 0 2024-09-15T20:38:44.628 INFO:tasks.workunit.client.0.smithi139.stdout:1.10 3 0 3 0 0 1851 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.397764+0000 45'30 87:120 [0,2] 0 [0,2] 0 38'29 2024-09-15T20:36:39.578100+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:37:31.289109+0000 2 0 2024-09-15T20:38:44.628 INFO:tasks.workunit.client.0.smithi139.stdout:1.11 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.398513+0000 37'29 87:90 [0,2] 0 [0,2] 0 36'28 2024-09-15T20:36:36.444590+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:48:59.386094+0000 2 0 2024-09-15T20:38:44.628 INFO:tasks.workunit.client.0.smithi139.stdout:1.12 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.397545+0000 36'28 87:105 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:35.623615+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:48:18.440517+0000 0 0 2024-09-15T20:38:44.628 INFO:tasks.workunit.client.0.smithi139.stdout:1.13 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.396874+0000 45'30 87:108 [0,2] 0 [0,2] 0 45'30 2024-09-15T20:36:55.766014+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:38:28.994911+0000 0 0 2024-09-15T20:38:44.628 INFO:tasks.workunit.client.0.smithi139.stdout:1.14 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:29.122393+0000 45'29 87:114 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:37.554032+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:59:56.311369+0000 1 0 2024-09-15T20:38:44.628 INFO:tasks.workunit.client.0.smithi139.stdout:1.15 3 0 3 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.411771+0000 41'29 87:105 [0,2] 0 [0,2] 0 41'29 2024-09-15T20:36:41.506894+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:44.628 INFO:tasks.workunit.client.0.smithi139.stdout:1.16 4 0 4 0 0 2468 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:28.410181+0000 41'31 87:115 [0,2] 0 [0,2] 0 41'31 2024-09-15T20:36:47.576518+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:29:37.611785+0000 3 0 2024-09-15T20:38:44.629 INFO:tasks.workunit.client.0.smithi139.stdout:1.17 1 0 1 0 0 520 0 0 15 12 15 active+undersized+degraded 2024-09-15T20:37:28.409266+0000 32'27 87:68 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:42.639678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:44.629 INFO:tasks.workunit.client.0.smithi139.stdout:1.18 3 0 3 0 0 1851 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.394810+0000 45'30 87:119 [2,0] 2 [2,0] 2 34'28 2024-09-15T20:36:36.622678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:29:24.808950+0000 1 0 2024-09-15T20:38:44.629 INFO:tasks.workunit.client.0.smithi139.stdout:1.19 2 0 2 0 0 1137 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.398489+0000 34'28 87:81 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:52.759747+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:44.629 INFO:tasks.workunit.client.0.smithi139.stdout:1.1a 4 0 4 0 0 2468 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.369086+0000 44'30 87:116 [2,0] 2 [2,0] 2 44'30 2024-09-15T20:36:53.716793+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:57:48.429965+0000 3 0 2024-09-15T20:38:44.629 INFO:tasks.workunit.client.0.smithi139.stdout:1.1b 5 0 5 0 0 3085 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:28.396910+0000 45'31 87:124 [0,2] 0 [0,2] 0 41'30 2024-09-15T20:36:41.658323+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:57:15.784191+0000 4 0 2024-09-15T20:38:44.629 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:38:44.629 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:38:44.629 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:38:44.641 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: ceph osd set noup 2024-09-15T20:38:45.313 INFO:tasks.workunit.client.0.smithi139.stderr:noup is set 2024-09-15T20:38:45.334 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-09-15T20:38:45.334 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-15T20:38:45.335 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:38:45.335 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:38:45.335 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:38:45.335 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-15T20:38:45.335 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:38:45.335 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:38:45.335 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:38:45.335 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:38:45.335 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T20:38:45.335 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T20:38:45.335 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:38:45.335 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:38:45.335 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:38:45.336 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:38:45.336 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:38:45.336 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:38:45.337 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:38:45.337 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:38:45.337 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:38:45.338 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:38:45.338 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:38:45.338 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:38:45.338 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:38:45.338 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:38:45.338 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:38:45.338 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:38:45.338 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:38:45.338 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:38:45.338 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:38:45.338 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:38:45.338 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-15T20:38:45.339 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.1 2024-09-15T20:38:45.339 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:38:45.339 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:38:45.339 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-15T20:38:45.340 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:38:45.340 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:38:45.340 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:38:45.340 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:38:45.378 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:38:45.378+0000 7f0d9676f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:38:45.381 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:38:45.378+0000 7f0d9676f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:38:45.382 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:38:45.382+0000 7f0d9676f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:38:45.863 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-15T20:38:45.863 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: sleep 5 2024-09-15T20:38:46.388 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:38:46.386+0000 7f0d9676f7c0 -1 Falling back to public interface 2024-09-15T20:38:49.573 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:38:49.574+0000 7f0d9676f7c0 -1 osd.1 64 log_to_monitors true 2024-09-15T20:38:50.864 INFO:tasks.workunit.client.0.smithi139.stdout:delay recovery divergent 2024-09-15T20:38:50.865 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:775: TEST_divergent_3: echo 'delay recovery divergent' 2024-09-15T20:38:50.865 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:776: TEST_divergent_3: ceph pg dump pgs 2024-09-15T20:38:51.297 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:38:51.297 INFO:tasks.workunit.client.0.smithi139.stdout:1.1f 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394529+0000 41'29 87:103 [2,0] 2 [2,0] 2 37'28 2024-09-15T20:36:40.657710+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:26:32.119186+0000 2 0 2024-09-15T20:38:51.297 INFO:tasks.workunit.client.0.smithi139.stdout:1.1e 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394381+0000 46'29 87:116 [2,0] 2 [2,0] 2 40'28 2024-09-15T20:36:45.584734+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:06:07.931182+0000 2 0 2024-09-15T20:38:51.297 INFO:tasks.workunit.client.0.smithi139.stdout:1.1d 4 0 4 0 0 2468 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:29.424550+0000 42'31 87:114 [2,0] 2 [2,0] 2 36'29 2024-09-15T20:36:37.493943+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:34:28.615082+0000 2 0 2024-09-15T20:38:51.297 INFO:tasks.workunit.client.0.smithi139.stdout:1.b 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.396794+0000 45'29 89:119 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:44.549410+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:27:32.595447+0000 2 0 2024-09-15T20:38:51.297 INFO:tasks.workunit.client.0.smithi139.stdout:1.a 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.368887+0000 39'29 87:93 [2,0] 2 [2,0] 2 39'29 2024-09-15T20:36:56.804751+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:38:28.674672+0000 0 0 2024-09-15T20:38:51.297 INFO:tasks.workunit.client.0.smithi139.stdout:1.9 5 0 5 0 0 3085 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.368691+0000 45'29 87:118 [2,0] 2 [2,0] 2 45'29 2024-09-15T20:36:51.656322+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:14:38.500654+0000 3 0 2024-09-15T20:38:51.297 INFO:tasks.workunit.client.0.smithi139.stdout:1.8 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.499768+0000 45'30 87:106 [2,0] 2 [2,0] 2 45'30 2024-09-15T20:36:50.633607+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:51.297 INFO:tasks.workunit.client.0.smithi139.stdout:1.7 2 0 2 0 0 1138 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:29.424254+0000 38'28 87:80 [2,0] 2 [2,0] 2 38'28 2024-09-15T20:36:48.576911+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:51.297 INFO:tasks.workunit.client.0.smithi139.stdout:1.6 4 0 4 0 0 2468 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.410719+0000 42'30 89:106 [0,2] 0 [0,2] 0 42'30 2024-09-15T20:36:46.625811+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:02:22.839337+0000 2 0 2024-09-15T20:38:51.298 INFO:tasks.workunit.client.0.smithi139.stdout:1.5 6 0 6 0 0 3702 0 0 20 12 20 active+undersized+degraded 2024-09-15T20:37:28.369909+0000 45'32 87:127 [2,0] 2 [2,0] 2 40'30 2024-09-15T20:36:40.468115+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:22:28.237620+0000 4 0 2024-09-15T20:38:51.298 INFO:tasks.workunit.client.0.smithi139.stdout:1.4 2 0 2 0 0 1234 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.399006+0000 40'28 89:108 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:38.561548+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:24:41.193931+0000 1 0 2024-09-15T20:38:51.298 INFO:tasks.workunit.client.0.smithi139.stdout:1.3 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.046599+0000 47'30 87:108 [2,0] 2 [2,0] 2 47'30 2024-09-15T20:36:54.739486+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:38:51.298 INFO:tasks.workunit.client.0.smithi139.stdout:1.2 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.409088+0000 42'28 89:103 [0,2] 0 [0,2] 0 42'28 2024-09-15T20:36:44.289898+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:59:17.740071+0000 1 0 2024-09-15T20:38:51.298 INFO:tasks.workunit.client.0.smithi139.stdout:1.c 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.395833+0000 44'29 89:120 [0,2] 0 [0,2] 0 38'28 2024-09-15T20:36:43.561129+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:59:06.361170+0000 2 0 2024-09-15T20:38:51.298 INFO:tasks.workunit.client.0.smithi139.stdout:1.1 2 0 2 0 0 1234 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:29.499685+0000 44'29 87:104 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:42.549754+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:03:26.939083+0000 0 0 2024-09-15T20:38:51.298 INFO:tasks.workunit.client.0.smithi139.stdout:1.1c 2 0 2 0 0 1234 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.398522+0000 36'29 89:99 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:34.684895+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:11:58.732534+0000 1 0 2024-09-15T20:38:51.298 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.409170+0000 45'29 89:123 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:31.575682+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:19:06.387325+0000 1 0 2024-09-15T20:38:51.298 INFO:tasks.workunit.client.0.smithi139.stdout:1.d 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394417+0000 45'29 87:113 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:39.548519+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:37:36.633002+0000 1 0 2024-09-15T20:38:51.298 INFO:tasks.workunit.client.0.smithi139.stdout:1.e 3 0 3 0 0 2105 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.097547+0000 87'30 87:94 [2,0] 2 [2,0] 2 44'29 2024-09-15T20:36:49.578000+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:38:30.733965+0000 0 0 2024-09-15T20:38:51.298 INFO:tasks.workunit.client.0.smithi139.stdout:1.f 4 0 4 0 0 2468 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.396239+0000 41'29 87:103 [2,0] 2 [2,0] 2 34'28 2024-09-15T20:36:32.588850+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:09:31.037907+0000 3 0 2024-09-15T20:38:51.298 INFO:tasks.workunit.client.0.smithi139.stdout:1.10 3 0 3 0 0 1851 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.397764+0000 45'30 89:124 [0,2] 0 [0,2] 0 38'29 2024-09-15T20:36:39.578100+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:37:31.289109+0000 2 0 2024-09-15T20:38:51.299 INFO:tasks.workunit.client.0.smithi139.stdout:1.11 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.398513+0000 37'29 89:94 [0,2] 0 [0,2] 0 36'28 2024-09-15T20:36:36.444590+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:48:59.386094+0000 2 0 2024-09-15T20:38:51.299 INFO:tasks.workunit.client.0.smithi139.stdout:1.12 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.397545+0000 36'28 89:109 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:35.623615+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:48:18.440517+0000 0 0 2024-09-15T20:38:51.299 INFO:tasks.workunit.client.0.smithi139.stdout:1.13 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.396874+0000 45'30 89:112 [0,2] 0 [0,2] 0 45'30 2024-09-15T20:36:55.766014+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:29.839934+0000 0 0 2024-09-15T20:38:51.299 INFO:tasks.workunit.client.0.smithi139.stdout:1.14 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:29.122393+0000 45'29 87:114 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:37.554032+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:59:56.311369+0000 1 0 2024-09-15T20:38:51.299 INFO:tasks.workunit.client.0.smithi139.stdout:1.15 3 0 3 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.411771+0000 41'29 89:109 [0,2] 0 [0,2] 0 41'29 2024-09-15T20:36:41.506894+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:32.904014+0000 0 0 2024-09-15T20:38:51.299 INFO:tasks.workunit.client.0.smithi139.stdout:1.16 4 0 4 0 0 2468 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:28.410181+0000 41'31 89:119 [0,2] 0 [0,2] 0 41'31 2024-09-15T20:36:47.576518+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:29:37.611785+0000 3 0 2024-09-15T20:38:51.299 INFO:tasks.workunit.client.0.smithi139.stdout:1.17 1 0 1 0 0 520 0 0 15 12 15 active+undersized+degraded 2024-09-15T20:37:28.409266+0000 32'27 89:72 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:42.639678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:31.888786+0000 0 0 2024-09-15T20:38:51.299 INFO:tasks.workunit.client.0.smithi139.stdout:1.18 3 0 3 0 0 1851 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.394810+0000 45'30 87:119 [2,0] 2 [2,0] 2 34'28 2024-09-15T20:36:36.622678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:29:24.808950+0000 1 0 2024-09-15T20:38:51.299 INFO:tasks.workunit.client.0.smithi139.stdout:1.19 2 0 2 0 0 1137 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.398489+0000 34'28 89:85 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:52.759747+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:30.846469+0000 0 0 2024-09-15T20:38:51.299 INFO:tasks.workunit.client.0.smithi139.stdout:1.1a 4 0 4 0 0 2468 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.369086+0000 44'30 87:116 [2,0] 2 [2,0] 2 44'30 2024-09-15T20:36:53.716793+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:57:48.429965+0000 3 0 2024-09-15T20:38:51.299 INFO:tasks.workunit.client.0.smithi139.stdout:1.1b 5 0 5 0 0 3085 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:28.396910+0000 45'31 89:128 [0,2] 0 [0,2] 0 41'30 2024-09-15T20:36:41.658323+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:57:15.784191+0000 4 0 2024-09-15T20:38:51.299 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:38:51.299 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:38:51.300 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:38:51.314 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: get_asok_path osd.1 2024-09-15T20:38:51.314 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T20:38:51.314 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T20:38:51.314 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:38:51.314 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:38:51.314 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:38:51.314 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.1.asok 2024-09-15T20:38:51.314 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: CEPH_ARGS= 2024-09-15T20:38:51.314 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.1.asok set_recovery_delay 100000 2024-09-15T20:38:51.425 INFO:tasks.workunit.client.0.smithi139.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T20:38:51.438 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: ceph osd unset noup 2024-09-15T20:38:52.295 INFO:tasks.workunit.client.0.smithi139.stderr:noup is unset 2024-09-15T20:38:52.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:781: TEST_divergent_3: wait_for_osd up 0 2024-09-15T20:38:52.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:38:52.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:38:52.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:38:52.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:38:52.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:52.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:38:52.320 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:38:52.320 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:52.320 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:38:52.833 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 76 up_thru 83 down_at 66 last_clean_interval [5,54) [v2:127.0.0.1:6802/3684664553,v1:127.0.0.1:6803/3684664553] [v2:127.0.0.1:6804/3684664553,v1:127.0.0.1:6805/3684664553] exists,up 8ceaad64-e87a-460e-8e98-3caf0f7eff12 2024-09-15T20:38:52.835 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:38:52.835 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:38:52.835 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:38:52.835 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: wait_for_osd up 1 2024-09-15T20:38:52.835 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:38:52.835 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:38:52.835 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:38:52.835 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:38:52.835 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:52.835 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:38:52.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:38:52.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:52.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:38:53.398 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 92 up_thru 92 down_at 67 last_clean_interval [10,64) [v2:127.0.0.1:6818/2815298135,v1:127.0.0.1:6819/2815298135] [v2:127.0.0.1:6820/2815298135,v1:127.0.0.1:6821/2815298135] exists,up d5b37d03-d9a9-4f3f-8c3e-d61cdbdce38e 2024-09-15T20:38:53.399 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:38:53.399 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:38:53.399 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:38:53.400 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: wait_for_osd up 2 2024-09-15T20:38:53.400 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:38:53.400 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:38:53.400 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:38:53.400 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:38:53.400 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:38:53.400 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:38:53.400 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:38:53.401 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:38:53.401 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:38:54.060 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up in weight 1 up_from 80 up_thru 92 down_at 68 last_clean_interval [15,53) [v2:127.0.0.1:6810/3676612316,v1:127.0.0.1:6811/3676612316] [v2:127.0.0.1:6812/3676612316,v1:127.0.0.1:6813/3676612316] exists,up 5347fe2a-12cd-44fe-9cd9-0dbc2bc7ebb8 2024-09-15T20:38:54.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:38:54.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:38:54.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:38:54.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:785: TEST_divergent_3: ceph pg dump pgs 2024-09-15T20:38:54.493 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:38:54.494 INFO:tasks.workunit.client.0.smithi139.stdout:1.1f 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394529+0000 41'29 90:108 [2,0] 2 [2,0] 2 37'28 2024-09-15T20:36:40.657710+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:26:32.119186+0000 2 0 2024-09-15T20:38:54.494 INFO:tasks.workunit.client.0.smithi139.stdout:1.1e 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394381+0000 46'29 90:121 [2,0] 2 [2,0] 2 40'28 2024-09-15T20:36:45.584734+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:06:07.931182+0000 2 0 2024-09-15T20:38:54.494 INFO:tasks.workunit.client.0.smithi139.stdout:1.1d 4 0 4 0 0 2468 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:29.424550+0000 42'31 90:119 [2,0] 2 [2,0] 2 36'29 2024-09-15T20:36:37.493943+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:34:28.615082+0000 2 0 2024-09-15T20:38:54.494 INFO:tasks.workunit.client.0.smithi139.stdout:1.b 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.396794+0000 45'29 90:121 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:44.549410+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:27:32.595447+0000 2 0 2024-09-15T20:38:54.494 INFO:tasks.workunit.client.0.smithi139.stdout:1.a 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.368887+0000 39'29 90:98 [2,0] 2 [2,0] 2 39'29 2024-09-15T20:36:56.804751+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:28.744130+0000 0 0 2024-09-15T20:38:54.494 INFO:tasks.workunit.client.0.smithi139.stdout:1.9 5 0 5 0 0 3085 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.368691+0000 45'29 90:123 [2,0] 2 [2,0] 2 45'29 2024-09-15T20:36:51.656322+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:14:38.500654+0000 3 0 2024-09-15T20:38:54.494 INFO:tasks.workunit.client.0.smithi139.stdout:1.8 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.499768+0000 45'30 90:111 [2,0] 2 [2,0] 2 45'30 2024-09-15T20:36:50.633607+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:33.799932+0000 0 0 2024-09-15T20:38:54.494 INFO:tasks.workunit.client.0.smithi139.stdout:1.7 2 0 2 0 0 1138 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:29.424254+0000 38'28 90:85 [2,0] 2 [2,0] 2 38'28 2024-09-15T20:36:48.576911+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:31.770165+0000 0 0 2024-09-15T20:38:54.494 INFO:tasks.workunit.client.0.smithi139.stdout:1.6 4 0 4 0 0 2468 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.410719+0000 42'30 90:108 [0,2] 0 [0,2] 0 42'30 2024-09-15T20:36:46.625811+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:02:22.839337+0000 2 0 2024-09-15T20:38:54.494 INFO:tasks.workunit.client.0.smithi139.stdout:1.5 6 0 6 0 0 3702 0 0 20 12 20 active+undersized+degraded 2024-09-15T20:37:28.369909+0000 45'32 90:132 [2,0] 2 [2,0] 2 40'30 2024-09-15T20:36:40.468115+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:22:28.237620+0000 4 0 2024-09-15T20:38:54.494 INFO:tasks.workunit.client.0.smithi139.stdout:1.4 2 0 2 0 0 1234 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.399006+0000 40'28 90:110 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:38.561548+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:24:41.193931+0000 1 0 2024-09-15T20:38:54.494 INFO:tasks.workunit.client.0.smithi139.stdout:1.3 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.046599+0000 47'30 90:113 [2,0] 2 [2,0] 2 47'30 2024-09-15T20:36:54.739486+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:29.763484+0000 0 0 2024-09-15T20:38:54.495 INFO:tasks.workunit.client.0.smithi139.stdout:1.2 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.409088+0000 42'28 90:105 [0,2] 0 [0,2] 0 42'28 2024-09-15T20:36:44.289898+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:59:17.740071+0000 1 0 2024-09-15T20:38:54.495 INFO:tasks.workunit.client.0.smithi139.stdout:1.c 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.395833+0000 44'29 90:122 [0,2] 0 [0,2] 0 38'28 2024-09-15T20:36:43.561129+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:59:06.361170+0000 2 0 2024-09-15T20:38:54.495 INFO:tasks.workunit.client.0.smithi139.stdout:1.1 2 0 2 0 0 1234 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:29.499685+0000 44'29 90:109 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:42.549754+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:03:26.939083+0000 0 0 2024-09-15T20:38:54.495 INFO:tasks.workunit.client.0.smithi139.stdout:1.1c 2 0 2 0 0 1234 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.398522+0000 36'29 90:101 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:34.684895+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:11:58.732534+0000 1 0 2024-09-15T20:38:54.495 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.409170+0000 45'29 90:125 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:31.575682+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:19:06.387325+0000 1 0 2024-09-15T20:38:54.495 INFO:tasks.workunit.client.0.smithi139.stdout:1.d 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394417+0000 45'29 90:118 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:39.548519+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:37:36.633002+0000 1 0 2024-09-15T20:38:54.495 INFO:tasks.workunit.client.0.smithi139.stdout:1.e 3 0 3 0 0 2105 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.097547+0000 87'30 90:99 [2,0] 2 [2,0] 2 44'29 2024-09-15T20:36:49.578000+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:30.761980+0000 0 0 2024-09-15T20:38:54.495 INFO:tasks.workunit.client.0.smithi139.stdout:1.f 4 0 4 0 0 2468 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.396239+0000 41'29 90:108 [2,0] 2 [2,0] 2 34'28 2024-09-15T20:36:32.588850+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:09:31.037907+0000 3 0 2024-09-15T20:38:54.495 INFO:tasks.workunit.client.0.smithi139.stdout:1.10 3 0 3 0 0 1851 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.397764+0000 45'30 90:126 [0,2] 0 [0,2] 0 38'29 2024-09-15T20:36:39.578100+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:37:31.289109+0000 2 0 2024-09-15T20:38:54.495 INFO:tasks.workunit.client.0.smithi139.stdout:1.11 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.398513+0000 37'29 90:96 [0,2] 0 [0,2] 0 36'28 2024-09-15T20:36:36.444590+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:48:59.386094+0000 2 0 2024-09-15T20:38:54.495 INFO:tasks.workunit.client.0.smithi139.stdout:1.12 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.397545+0000 36'28 90:111 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:35.623615+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:48:18.440517+0000 0 0 2024-09-15T20:38:54.495 INFO:tasks.workunit.client.0.smithi139.stdout:1.13 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.396874+0000 45'30 90:114 [0,2] 0 [0,2] 0 45'30 2024-09-15T20:36:55.766014+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:29.839934+0000 0 0 2024-09-15T20:38:54.495 INFO:tasks.workunit.client.0.smithi139.stdout:1.14 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:29.122393+0000 45'29 90:119 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:37.554032+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:59:56.311369+0000 1 0 2024-09-15T20:38:54.496 INFO:tasks.workunit.client.0.smithi139.stdout:1.15 3 0 3 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.411771+0000 41'29 90:111 [0,2] 0 [0,2] 0 41'29 2024-09-15T20:36:41.506894+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:32.904014+0000 0 0 2024-09-15T20:38:54.496 INFO:tasks.workunit.client.0.smithi139.stdout:1.16 4 0 4 0 0 2468 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:28.410181+0000 41'31 90:121 [0,2] 0 [0,2] 0 41'31 2024-09-15T20:36:47.576518+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:29:37.611785+0000 3 0 2024-09-15T20:38:54.496 INFO:tasks.workunit.client.0.smithi139.stdout:1.17 1 0 1 0 0 520 0 0 15 12 15 active+undersized+degraded 2024-09-15T20:37:28.409266+0000 32'27 90:74 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:42.639678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:31.888786+0000 0 0 2024-09-15T20:38:54.496 INFO:tasks.workunit.client.0.smithi139.stdout:1.18 3 0 3 0 0 1851 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.394810+0000 45'30 90:124 [2,0] 2 [2,0] 2 34'28 2024-09-15T20:36:36.622678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:29:24.808950+0000 1 0 2024-09-15T20:38:54.496 INFO:tasks.workunit.client.0.smithi139.stdout:1.19 2 0 2 0 0 1137 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.398489+0000 34'28 90:87 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:52.759747+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:30.846469+0000 0 0 2024-09-15T20:38:54.496 INFO:tasks.workunit.client.0.smithi139.stdout:1.1a 4 0 4 0 0 2468 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.369086+0000 44'30 90:121 [2,0] 2 [2,0] 2 44'30 2024-09-15T20:36:53.716793+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:57:48.429965+0000 3 0 2024-09-15T20:38:54.496 INFO:tasks.workunit.client.0.smithi139.stdout:1.1b 5 0 5 0 0 3085 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:28.396910+0000 45'31 90:130 [0,2] 0 [0,2] 0 41'30 2024-09-15T20:36:41.658323+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:57:15.784191+0000 4 0 2024-09-15T20:38:54.496 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:38:54.496 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:38:54.496 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:38:54.511 INFO:tasks.workunit.client.0.smithi139.stdout:wait for peering 2024-09-15T20:38:54.511 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'wait for peering' 2024-09-15T20:38:54.511 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2024-09-15T20:38:54.983 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:38:54.983 INFO:tasks.workunit.client.0.smithi139.stdout:1.1f 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394529+0000 41'29 90:108 [2,0] 2 [2,0] 2 37'28 2024-09-15T20:36:40.657710+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:26:32.119186+0000 2 0 2024-09-15T20:38:54.983 INFO:tasks.workunit.client.0.smithi139.stdout:1.1e 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394381+0000 46'29 90:121 [2,0] 2 [2,0] 2 40'28 2024-09-15T20:36:45.584734+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:06:07.931182+0000 2 0 2024-09-15T20:38:54.984 INFO:tasks.workunit.client.0.smithi139.stdout:1.1d 4 0 4 0 0 2468 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:29.424550+0000 42'31 90:119 [2,0] 2 [2,0] 2 36'29 2024-09-15T20:36:37.493943+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:34:28.615082+0000 2 0 2024-09-15T20:38:54.984 INFO:tasks.workunit.client.0.smithi139.stdout:1.b 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.396794+0000 45'29 90:121 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:44.549410+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:27:32.595447+0000 2 0 2024-09-15T20:38:54.984 INFO:tasks.workunit.client.0.smithi139.stdout:1.a 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.368887+0000 39'29 90:98 [2,0] 2 [2,0] 2 39'29 2024-09-15T20:36:56.804751+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:28.744130+0000 0 0 2024-09-15T20:38:54.984 INFO:tasks.workunit.client.0.smithi139.stdout:1.9 5 0 5 0 0 3085 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.368691+0000 45'29 90:123 [2,0] 2 [2,0] 2 45'29 2024-09-15T20:36:51.656322+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:14:38.500654+0000 3 0 2024-09-15T20:38:54.984 INFO:tasks.workunit.client.0.smithi139.stdout:1.8 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.499768+0000 45'30 90:111 [2,0] 2 [2,0] 2 45'30 2024-09-15T20:36:50.633607+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:33.799932+0000 0 0 2024-09-15T20:38:54.984 INFO:tasks.workunit.client.0.smithi139.stdout:1.7 2 0 2 0 0 1138 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:29.424254+0000 38'28 90:85 [2,0] 2 [2,0] 2 38'28 2024-09-15T20:36:48.576911+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:31.770165+0000 0 0 2024-09-15T20:38:54.984 INFO:tasks.workunit.client.0.smithi139.stdout:1.6 4 0 4 0 0 2468 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.410719+0000 42'30 90:108 [0,2] 0 [0,2] 0 42'30 2024-09-15T20:36:46.625811+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:02:22.839337+0000 2 0 2024-09-15T20:38:54.984 INFO:tasks.workunit.client.0.smithi139.stdout:1.5 6 0 6 0 0 3702 0 0 20 12 20 active+undersized+degraded 2024-09-15T20:37:28.369909+0000 45'32 90:132 [2,0] 2 [2,0] 2 40'30 2024-09-15T20:36:40.468115+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:22:28.237620+0000 4 0 2024-09-15T20:38:54.984 INFO:tasks.workunit.client.0.smithi139.stdout:1.4 2 0 2 0 0 1234 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.399006+0000 40'28 90:110 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:38.561548+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:24:41.193931+0000 1 0 2024-09-15T20:38:54.984 INFO:tasks.workunit.client.0.smithi139.stdout:1.3 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.046599+0000 47'30 90:113 [2,0] 2 [2,0] 2 47'30 2024-09-15T20:36:54.739486+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:29.763484+0000 0 0 2024-09-15T20:38:54.984 INFO:tasks.workunit.client.0.smithi139.stdout:1.2 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.409088+0000 42'28 90:105 [0,2] 0 [0,2] 0 42'28 2024-09-15T20:36:44.289898+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:59:17.740071+0000 1 0 2024-09-15T20:38:54.985 INFO:tasks.workunit.client.0.smithi139.stdout:1.c 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.395833+0000 44'29 90:122 [0,2] 0 [0,2] 0 38'28 2024-09-15T20:36:43.561129+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:59:06.361170+0000 2 0 2024-09-15T20:38:54.985 INFO:tasks.workunit.client.0.smithi139.stdout:1.1 2 0 2 0 0 1234 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:29.499685+0000 44'29 90:109 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:42.549754+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:03:26.939083+0000 0 0 2024-09-15T20:38:54.985 INFO:tasks.workunit.client.0.smithi139.stdout:1.1c 2 0 2 0 0 1234 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.398522+0000 36'29 90:101 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:34.684895+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:11:58.732534+0000 1 0 2024-09-15T20:38:54.985 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.409170+0000 45'29 90:125 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:31.575682+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:19:06.387325+0000 1 0 2024-09-15T20:38:54.985 INFO:tasks.workunit.client.0.smithi139.stdout:1.d 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394417+0000 45'29 90:118 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:39.548519+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:37:36.633002+0000 1 0 2024-09-15T20:38:54.985 INFO:tasks.workunit.client.0.smithi139.stdout:1.e 3 0 3 0 0 2105 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.097547+0000 87'30 90:99 [2,0] 2 [2,0] 2 44'29 2024-09-15T20:36:49.578000+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:30.761980+0000 0 0 2024-09-15T20:38:54.985 INFO:tasks.workunit.client.0.smithi139.stdout:1.f 4 0 4 0 0 2468 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.396239+0000 41'29 90:108 [2,0] 2 [2,0] 2 34'28 2024-09-15T20:36:32.588850+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:09:31.037907+0000 3 0 2024-09-15T20:38:54.985 INFO:tasks.workunit.client.0.smithi139.stdout:1.10 3 0 3 0 0 1851 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.397764+0000 45'30 90:126 [0,2] 0 [0,2] 0 38'29 2024-09-15T20:36:39.578100+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:37:31.289109+0000 2 0 2024-09-15T20:38:54.985 INFO:tasks.workunit.client.0.smithi139.stdout:1.11 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.398513+0000 37'29 90:96 [0,2] 0 [0,2] 0 36'28 2024-09-15T20:36:36.444590+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:48:59.386094+0000 2 0 2024-09-15T20:38:54.985 INFO:tasks.workunit.client.0.smithi139.stdout:1.12 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.397545+0000 36'28 90:111 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:35.623615+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:48:18.440517+0000 0 0 2024-09-15T20:38:54.985 INFO:tasks.workunit.client.0.smithi139.stdout:1.13 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.396874+0000 45'30 90:114 [0,2] 0 [0,2] 0 45'30 2024-09-15T20:36:55.766014+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:29.839934+0000 0 0 2024-09-15T20:38:54.985 INFO:tasks.workunit.client.0.smithi139.stdout:1.14 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:29.122393+0000 45'29 90:119 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:37.554032+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:59:56.311369+0000 1 0 2024-09-15T20:38:54.985 INFO:tasks.workunit.client.0.smithi139.stdout:1.15 3 0 3 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.411771+0000 41'29 90:111 [0,2] 0 [0,2] 0 41'29 2024-09-15T20:36:41.506894+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:32.904014+0000 0 0 2024-09-15T20:38:54.986 INFO:tasks.workunit.client.0.smithi139.stdout:1.16 4 0 4 0 0 2468 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:28.410181+0000 41'31 90:121 [0,2] 0 [0,2] 0 41'31 2024-09-15T20:36:47.576518+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:29:37.611785+0000 3 0 2024-09-15T20:38:54.986 INFO:tasks.workunit.client.0.smithi139.stdout:1.17 1 0 1 0 0 520 0 0 15 12 15 active+undersized+degraded 2024-09-15T20:37:28.409266+0000 32'27 90:74 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:42.639678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:31.888786+0000 0 0 2024-09-15T20:38:54.986 INFO:tasks.workunit.client.0.smithi139.stdout:1.18 3 0 3 0 0 1851 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.394810+0000 45'30 90:124 [2,0] 2 [2,0] 2 34'28 2024-09-15T20:36:36.622678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:29:24.808950+0000 1 0 2024-09-15T20:38:54.986 INFO:tasks.workunit.client.0.smithi139.stdout:1.19 2 0 2 0 0 1137 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.398489+0000 34'28 90:87 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:52.759747+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:30.846469+0000 0 0 2024-09-15T20:38:54.986 INFO:tasks.workunit.client.0.smithi139.stdout:1.1a 4 0 4 0 0 2468 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.369086+0000 44'30 90:121 [2,0] 2 [2,0] 2 44'30 2024-09-15T20:36:53.716793+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:57:48.429965+0000 3 0 2024-09-15T20:38:54.986 INFO:tasks.workunit.client.0.smithi139.stdout:1.1b 5 0 5 0 0 3085 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:28.396910+0000 45'31 90:130 [0,2] 0 [0,2] 0 41'30 2024-09-15T20:36:41.658323+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:57:15.784191+0000 4 0 2024-09-15T20:38:54.986 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:38:54.986 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:38:54.986 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:38:55.000 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: rados -p test put foo /etc/fstab 2024-09-15T20:38:55.104 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: echo 'killing divergent 1' 2024-09-15T20:38:55.104 INFO:tasks.workunit.client.0.smithi139.stdout:killing divergent 1 2024-09-15T20:38:55.104 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:793: TEST_divergent_3: ceph pg dump pgs 2024-09-15T20:38:55.534 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:38:55.534 INFO:tasks.workunit.client.0.smithi139.stdout:1.1f 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394529+0000 41'29 90:108 [2,0] 2 [2,0] 2 37'28 2024-09-15T20:36:40.657710+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:26:32.119186+0000 2 0 2024-09-15T20:38:55.534 INFO:tasks.workunit.client.0.smithi139.stdout:1.1e 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394381+0000 46'29 90:121 [2,0] 2 [2,0] 2 40'28 2024-09-15T20:36:45.584734+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:06:07.931182+0000 2 0 2024-09-15T20:38:55.534 INFO:tasks.workunit.client.0.smithi139.stdout:1.1d 4 0 4 0 0 2468 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:29.424550+0000 42'31 90:119 [2,0] 2 [2,0] 2 36'29 2024-09-15T20:36:37.493943+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:34:28.615082+0000 2 0 2024-09-15T20:38:55.535 INFO:tasks.workunit.client.0.smithi139.stdout:1.b 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.396794+0000 45'29 90:121 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:44.549410+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:27:32.595447+0000 2 0 2024-09-15T20:38:55.535 INFO:tasks.workunit.client.0.smithi139.stdout:1.a 3 0 3 0 0 1755 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.368887+0000 39'29 90:98 [2,0] 2 [2,0] 2 39'29 2024-09-15T20:36:56.804751+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:28.744130+0000 0 0 2024-09-15T20:38:55.535 INFO:tasks.workunit.client.0.smithi139.stdout:1.9 5 0 5 0 0 3085 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.368691+0000 45'29 90:123 [2,0] 2 [2,0] 2 45'29 2024-09-15T20:36:51.656322+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:14:38.500654+0000 3 0 2024-09-15T20:38:55.535 INFO:tasks.workunit.client.0.smithi139.stdout:1.8 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.499768+0000 45'30 90:111 [2,0] 2 [2,0] 2 45'30 2024-09-15T20:36:50.633607+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:33.799932+0000 0 0 2024-09-15T20:38:55.535 INFO:tasks.workunit.client.0.smithi139.stdout:1.7 2 0 2 0 0 1138 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:29.424254+0000 38'28 90:85 [2,0] 2 [2,0] 2 38'28 2024-09-15T20:36:48.576911+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:31.770165+0000 0 0 2024-09-15T20:38:55.535 INFO:tasks.workunit.client.0.smithi139.stdout:1.6 4 0 4 0 0 2468 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.410719+0000 42'30 90:108 [0,2] 0 [0,2] 0 42'30 2024-09-15T20:36:46.625811+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:02:22.839337+0000 2 0 2024-09-15T20:38:55.535 INFO:tasks.workunit.client.0.smithi139.stdout:1.5 6 0 6 0 0 3702 0 0 20 12 20 active+undersized+degraded 2024-09-15T20:37:28.369909+0000 45'32 90:132 [2,0] 2 [2,0] 2 40'30 2024-09-15T20:36:40.468115+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:22:28.237620+0000 4 0 2024-09-15T20:38:55.535 INFO:tasks.workunit.client.0.smithi139.stdout:1.4 2 0 2 0 0 1234 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.399006+0000 40'28 90:110 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:38.561548+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:24:41.193931+0000 1 0 2024-09-15T20:38:55.535 INFO:tasks.workunit.client.0.smithi139.stdout:1.3 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.046599+0000 47'30 90:113 [2,0] 2 [2,0] 2 47'30 2024-09-15T20:36:54.739486+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:29.763484+0000 0 0 2024-09-15T20:38:55.535 INFO:tasks.workunit.client.0.smithi139.stdout:1.2 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.409088+0000 42'28 90:105 [0,2] 0 [0,2] 0 42'28 2024-09-15T20:36:44.289898+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:59:17.740071+0000 1 0 2024-09-15T20:38:55.535 INFO:tasks.workunit.client.0.smithi139.stdout:1.c 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.395833+0000 44'29 90:122 [0,2] 0 [0,2] 0 38'28 2024-09-15T20:36:43.561129+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:59:06.361170+0000 2 0 2024-09-15T20:38:55.535 INFO:tasks.workunit.client.0.smithi139.stdout:1.1 2 0 2 0 0 1234 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:29.499685+0000 44'29 90:109 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:42.549754+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:03:26.939083+0000 0 0 2024-09-15T20:38:55.536 INFO:tasks.workunit.client.0.smithi139.stdout:1.1c 2 0 2 0 0 1234 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.398522+0000 36'29 90:101 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:34.684895+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:11:58.732534+0000 1 0 2024-09-15T20:38:55.536 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.409170+0000 45'29 90:125 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:31.575682+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:19:06.387325+0000 1 0 2024-09-15T20:38:55.536 INFO:tasks.workunit.client.0.smithi139.stdout:1.d 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.394417+0000 45'29 90:118 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:39.548519+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:37:36.633002+0000 1 0 2024-09-15T20:38:55.536 INFO:tasks.workunit.client.0.smithi139.stdout:1.e 3 0 3 0 0 2105 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:29.097547+0000 87'30 90:99 [2,0] 2 [2,0] 2 44'29 2024-09-15T20:36:49.578000+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:30.761980+0000 0 0 2024-09-15T20:38:55.536 INFO:tasks.workunit.client.0.smithi139.stdout:1.f 4 0 4 0 0 2468 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.396239+0000 41'29 90:108 [2,0] 2 [2,0] 2 34'28 2024-09-15T20:36:32.588850+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:09:31.037907+0000 3 0 2024-09-15T20:38:55.536 INFO:tasks.workunit.client.0.smithi139.stdout:1.10 3 0 3 0 0 1851 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.397764+0000 45'30 90:126 [0,2] 0 [0,2] 0 38'29 2024-09-15T20:36:39.578100+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:37:31.289109+0000 2 0 2024-09-15T20:38:55.536 INFO:tasks.workunit.client.0.smithi139.stdout:1.11 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.398513+0000 37'29 90:96 [0,2] 0 [0,2] 0 36'28 2024-09-15T20:36:36.444590+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:48:59.386094+0000 2 0 2024-09-15T20:38:55.536 INFO:tasks.workunit.client.0.smithi139.stdout:1.12 1 0 1 0 0 617 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.397545+0000 36'28 90:111 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:35.623615+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:48:18.440517+0000 0 0 2024-09-15T20:38:55.536 INFO:tasks.workunit.client.0.smithi139.stdout:1.13 4 0 4 0 0 2372 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.396874+0000 45'30 90:114 [0,2] 0 [0,2] 0 45'30 2024-09-15T20:36:55.766014+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:29.839934+0000 0 0 2024-09-15T20:38:55.536 INFO:tasks.workunit.client.0.smithi139.stdout:1.14 3 0 3 0 0 1851 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:29.122393+0000 45'29 90:119 [2,0] 2 [2,0] 2 32'27 2024-09-15T20:36:37.554032+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:59:56.311369+0000 1 0 2024-09-15T20:38:55.536 INFO:tasks.workunit.client.0.smithi139.stdout:1.15 3 0 3 0 0 1754 0 0 17 12 17 active+undersized+degraded 2024-09-15T20:37:28.411771+0000 41'29 90:111 [0,2] 0 [0,2] 0 41'29 2024-09-15T20:36:41.506894+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:32.904014+0000 0 0 2024-09-15T20:38:55.536 INFO:tasks.workunit.client.0.smithi139.stdout:1.16 4 0 4 0 0 2468 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:28.410181+0000 41'31 90:121 [0,2] 0 [0,2] 0 41'31 2024-09-15T20:36:47.576518+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:29:37.611785+0000 3 0 2024-09-15T20:38:55.536 INFO:tasks.workunit.client.0.smithi139.stdout:1.17 1 0 1 0 0 520 0 0 15 12 15 active+undersized+degraded 2024-09-15T20:37:28.409266+0000 32'27 90:74 [0,2] 0 [0,2] 0 32'27 2024-09-15T20:36:42.639678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:31.888786+0000 0 0 2024-09-15T20:38:55.537 INFO:tasks.workunit.client.0.smithi139.stdout:1.18 3 0 3 0 0 1851 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.394810+0000 45'30 90:124 [2,0] 2 [2,0] 2 34'28 2024-09-15T20:36:36.622678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:29:24.808950+0000 1 0 2024-09-15T20:38:55.537 INFO:tasks.workunit.client.0.smithi139.stdout:1.19 2 0 2 0 0 1137 0 0 16 12 16 active+undersized+degraded 2024-09-15T20:37:28.398489+0000 34'28 90:87 [0,2] 0 [0,2] 0 34'28 2024-09-15T20:36:52.759747+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 user requested scrub scheduled @ 2024-09-15T20:39:30.846469+0000 0 0 2024-09-15T20:38:55.537 INFO:tasks.workunit.client.0.smithi139.stdout:1.1a 4 0 4 0 0 2468 0 0 18 12 18 active+undersized+degraded 2024-09-15T20:37:28.369086+0000 44'30 90:121 [2,0] 2 [2,0] 2 44'30 2024-09-15T20:36:53.716793+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:57:48.429965+0000 3 0 2024-09-15T20:38:55.537 INFO:tasks.workunit.client.0.smithi139.stdout:1.1b 5 0 5 0 0 3085 0 0 19 12 19 active+undersized+degraded 2024-09-15T20:37:28.396910+0000 45'31 90:130 [0,2] 0 [0,2] 0 41'30 2024-09-15T20:36:41.658323+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:57:15.784191+0000 4 0 2024-09-15T20:38:55.537 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:38:55.537 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:38:55.537 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:38:55.551 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2024-09-15T20:38:55.552 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:38:55.552 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:38:55.552 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:38:55.553 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:38:55.553 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:38:55.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:38:55.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: expfile=td/divergent-priors/exp.27419.out 2024-09-15T20:38:55.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:798: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.27419.out 2024-09-15T20:38:55.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-15T20:38:55.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:38:55.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:38:55.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:38:55.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-15T20:38:55.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.27419.out 2024-09-15T20:38:57.099 INFO:tasks.workunit.client.0.smithi139.stderr:Exporting 1.1b info 1.1b( v 45'31 lc 0'0 (31'12,45'31] local-lis/les=92/93 n=5 ec=33/23 lis/c=92/52 les/c/f=93/53/0 sis=92) 2024-09-15T20:38:57.100 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-09-15T20:38:57.109 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-09-15T20:38:57.110 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:dd677bd3:::existing_35:head# 2024-09-15T20:38:57.117 INFO:tasks.workunit.client.0.smithi139.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-09-15T20:38:57.117 INFO:tasks.workunit.client.0.smithi139.stderr:Export successful 2024-09-15T20:38:57.118 INFO:tasks.workunit.client.0.smithi139.stdout: marking collection for removal 2024-09-15T20:38:57.118 INFO:tasks.workunit.client.0.smithi139.stdout:setting '_remove' omap key 2024-09-15T20:38:57.118 INFO:tasks.workunit.client.0.smithi139.stdout:finish_remove_pgs 1.1b_head removing 1.1b 2024-09-15T20:38:57.215 INFO:tasks.workunit.client.0.smithi139.stderr:Remove successful 2024-09-15T20:38:57.659 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.27419.out 2024-09-15T20:38:57.659 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-15T20:38:57.659 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:38:57.660 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:38:57.660 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:38:57.660 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-15T20:38:57.660 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.27419.out 2024-09-15T20:38:58.881 INFO:tasks.workunit.client.0.smithi139.stdout:get_pg_num_history pg_num_history pg_num_history(e94 pg_nums {1={23=1,33=32}} deleted_pools ) 2024-09-15T20:38:58.881 INFO:tasks.workunit.client.0.smithi139.stdout:Importing pgid 1.1b 2024-09-15T20:38:58.881 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-09-15T20:38:58.881 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:38:58.935 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-09-15T20:38:58.935 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:38:58.960 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:dd677bd3:::existing_35:head# 2024-09-15T20:38:58.961 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:38:58.985 INFO:tasks.workunit.client.0.smithi139.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-09-15T20:38:58.985 INFO:tasks.workunit.client.0.smithi139.stdout:snapset 0=[]:{} 2024-09-15T20:38:59.010 INFO:tasks.workunit.client.0.smithi139.stdout:write_pg epoch 93 info 1.1b( v 45'31 lc 0'0 (31'12,45'31] local-lis/les=92/93 n=5 ec=33/23 lis/c=92/52 les/c/f=93/53/0 sis=92) 2024-09-15T20:38:59.035 INFO:tasks.workunit.client.0.smithi139.stdout:Import successful 2024-09-15T20:38:59.580 INFO:tasks.workunit.client.0.smithi139.stdout:reviving divergent 1 2024-09-15T20:38:59.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:801: TEST_divergent_3: echo 'reviving divergent 1' 2024-09-15T20:38:59.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: ceph pg dump pgs 2024-09-15T20:39:00.011 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:39:00.011 INFO:tasks.workunit.client.0.smithi139.stdout:1.1f 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:38:53.359795+0000 41'29 94:123 [2,0,1] 2 [2,0,1] 2 41'29 2024-09-15T20:37:07.497422+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:03:39.981785+0000 2 0 2024-09-15T20:39:00.011 INFO:tasks.workunit.client.0.smithi139.stdout:1.1e 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:38:53.433461+0000 46'29 94:153 [1,2,0] 1 [1,2,0] 1 40'28 2024-09-15T20:36:45.584734+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T23:46:49.197438+0000 2 0 2024-09-15T20:39:00.012 INFO:tasks.workunit.client.0.smithi139.stdout:1.1d 4 0 0 0 0 2468 0 0 19 12 19 active+clean 2024-09-15T20:38:53.360433+0000 42'31 94:134 [2,1,0] 2 [2,1,0] 2 36'29 2024-09-15T20:36:37.493943+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:03:23.462181+0000 2 0 2024-09-15T20:39:00.012 INFO:tasks.workunit.client.0.smithi139.stdout:1.b 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:38:53.430467+0000 45'29 94:141 [1,0,2] 1 [1,0,2] 1 34'28 2024-09-15T20:36:44.549410+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:40:05.262281+0000 2 0 2024-09-15T20:39:00.012 INFO:tasks.workunit.client.0.smithi139.stdout:1.a 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:38:53.434057+0000 39'29 94:111 [1,2,0] 1 [1,2,0] 1 39'29 2024-09-15T20:36:56.804751+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:35:47.823543+0000 3 0 2024-09-15T20:39:00.012 INFO:tasks.workunit.client.0.smithi139.stdout:1.9 5 1 1 0 0 3085 0 0 17 12 17 active+recovery_wait+degraded 2024-09-15T20:38:53.435361+0000 45'29 94:142 [1,2,0] 1 [1,2,0] 1 45'29 2024-09-15T20:36:51.656322+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 no scrub is scheduled 3 0 2024-09-15T20:39:00.012 INFO:tasks.workunit.client.0.smithi139.stdout:1.8 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-15T20:38:53.434855+0000 45'30 94:135 [1,2,0] 1 [1,2,0] 1 45'30 2024-09-15T20:36:50.633607+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:39:00.012 INFO:tasks.workunit.client.0.smithi139.stdout:1.7 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-15T20:38:54.554976+0000 38'28 94:126 [1,2,0] 1 [1,2,0] 1 38'28 2024-09-15T20:38:54.554790+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:09:14.833474+0000 2 0 2024-09-15T20:39:00.012 INFO:tasks.workunit.client.0.smithi139.stdout:1.6 4 0 0 0 0 2468 0 0 18 12 18 active+clean 2024-09-15T20:38:53.429570+0000 42'30 94:121 [1,0,2] 1 [1,0,2] 1 42'30 2024-09-15T20:36:46.625811+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T23:47:36.544278+0000 2 0 2024-09-15T20:39:00.012 INFO:tasks.workunit.client.0.smithi139.stdout:1.5 6 0 0 0 0 3702 0 0 20 12 20 active+clean 2024-09-15T20:38:53.363157+0000 45'32 94:147 [2,0,1] 2 [2,0,1] 2 40'30 2024-09-15T20:36:40.468115+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:27:30.594833+0000 4 0 2024-09-15T20:39:00.012 INFO:tasks.workunit.client.0.smithi139.stdout:1.4 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-15T20:38:53.434727+0000 40'28 94:104 [1,0,2] 1 [1,0,2] 1 32'27 2024-09-15T20:36:38.561548+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T05:32:14.865065+0000 1 0 2024-09-15T20:39:00.012 INFO:tasks.workunit.client.0.smithi139.stdout:1.3 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-15T20:38:53.434157+0000 47'30 94:130 [1,2,0] 1 [1,2,0] 1 47'30 2024-09-15T20:36:54.739486+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:39:00.012 INFO:tasks.workunit.client.0.smithi139.stdout:1.2 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-15T20:38:53.704904+0000 42'28 94:119 [0,1,2] 0 [0,1,2] 0 42'28 2024-09-15T20:36:44.289898+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 periodic scrub scheduled @ 2024-09-17T05:57:27.124432+0000 1 0 2024-09-15T20:39:00.012 INFO:tasks.workunit.client.0.smithi139.stdout:1.c 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:38:53.431209+0000 44'29 94:116 [1,0,2] 1 [1,0,2] 1 38'28 2024-09-15T20:36:43.561129+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T23:10:00.231367+0000 2 0 2024-09-15T20:39:00.012 INFO:tasks.workunit.client.0.smithi139.stdout:1.1 2 0 0 0 0 1234 0 0 17 12 17 active+clean 2024-09-15T20:38:53.360861+0000 44'29 94:124 [2,0,1] 2 [2,0,1] 2 32'27 2024-09-15T20:36:42.549754+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T23:58:40.336657+0000 0 0 2024-09-15T20:39:00.013 INFO:tasks.workunit.client.0.smithi139.stdout:1.1c 2 0 0 0 0 1234 0 0 17 12 17 active+clean 2024-09-15T20:38:53.435475+0000 36'29 94:95 [1,0,2] 1 [1,0,2] 1 34'28 2024-09-15T20:36:34.684895+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:52:06.218697+0000 1 0 2024-09-15T20:39:00.013 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:38:53.432849+0000 45'29 94:119 [1,0,2] 1 [1,0,2] 1 32'27 2024-09-15T20:36:31.575682+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:46:12.073226+0000 1 0 2024-09-15T20:39:00.013 INFO:tasks.workunit.client.0.smithi139.stdout:1.d 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:38:53.357911+0000 45'29 94:133 [2,1,0] 2 [2,1,0] 2 32'27 2024-09-15T20:36:39.548519+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:47:39.755482+0000 1 0 2024-09-15T20:39:00.013 INFO:tasks.workunit.client.0.smithi139.stdout:1.e 3 1 1 0 0 2105 0 0 18 12 18 active+recovering+degraded 2024-09-15T20:38:53.363600+0000 87'30 94:148 [1,2,0] 1 [1,2,0] 1 44'29 2024-09-15T20:36:49.578000+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 no scrub is scheduled 0 0 2024-09-15T20:39:00.013 INFO:tasks.workunit.client.0.smithi139.stdout:1.f 4 0 0 0 0 2468 0 0 17 12 17 active+clean 2024-09-15T20:38:53.362419+0000 41'29 94:123 [2,0,1] 2 [2,0,1] 2 41'29 2024-09-15T20:37:08.463113+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:58:28.891656+0000 3 0 2024-09-15T20:39:00.013 INFO:tasks.workunit.client.0.smithi139.stdout:1.10 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-15T20:38:53.767355+0000 45'30 94:140 [0,2,1] 0 [0,2,1] 0 38'29 2024-09-15T20:36:39.578100+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:07:41.781462+0000 2 0 2024-09-15T20:39:00.013 INFO:tasks.workunit.client.0.smithi139.stdout:1.11 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:38:53.768341+0000 37'29 94:111 [0,1,2] 0 [0,1,2] 0 36'28 2024-09-15T20:36:36.444590+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:38:24.307441+0000 2 0 2024-09-15T20:39:00.013 INFO:tasks.workunit.client.0.smithi139.stdout:1.12 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-15T20:38:53.770091+0000 36'28 94:125 [0,1,2] 0 [0,1,2] 0 32'27 2024-09-15T20:36:35.623615+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:35:56.762733+0000 0 0 2024-09-15T20:39:00.013 INFO:tasks.workunit.client.0.smithi139.stdout:1.13 4 0 0 0 0 2372 0 0 18 12 18 active+clean 2024-09-15T20:38:53.435066+0000 45'30 94:130 [1,0,2] 1 [1,0,2] 1 45'30 2024-09-15T20:36:55.766014+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:39:00.013 INFO:tasks.workunit.client.0.smithi139.stdout:1.14 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:38:53.361518+0000 45'29 94:134 [2,0,1] 2 [2,0,1] 2 32'27 2024-09-15T20:36:37.554032+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:04:52.139643+0000 1 0 2024-09-15T20:39:00.013 INFO:tasks.workunit.client.0.smithi139.stdout:1.15 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:38:53.664575+0000 41'29 94:112 [1,0,2] 1 [1,0,2] 1 41'29 2024-09-15T20:38:53.664450+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:50:43.674872+0000 3 0 2024-09-15T20:39:00.013 INFO:tasks.workunit.client.0.smithi139.stdout:1.16 4 0 0 0 0 2468 0 0 19 12 19 active+clean 2024-09-15T20:38:53.430480+0000 41'31 94:131 [1,0,2] 1 [1,0,2] 1 41'31 2024-09-15T20:36:47.576518+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:10:09.709405+0000 3 0 2024-09-15T20:39:00.014 INFO:tasks.workunit.client.0.smithi139.stdout:1.17 1 0 0 0 0 617 0 0 15 12 15 active+clean 2024-09-15T20:38:53.869627+0000 32'27 94:96 [0,1,2] 0 [0,1,2] 0 32'27 2024-09-15T20:38:53.869510+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:00:45.390297+0000 1 0 2024-09-15T20:39:00.014 INFO:tasks.workunit.client.0.smithi139.stdout:1.18 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-15T20:38:53.361337+0000 45'30 94:139 [2,1,0] 2 [2,1,0] 2 34'28 2024-09-15T20:36:36.622678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:56:42.660194+0000 1 0 2024-09-15T20:39:00.014 INFO:tasks.workunit.client.0.smithi139.stdout:1.19 2 0 0 0 0 1137 0 0 16 12 16 active+clean 2024-09-15T20:38:53.434594+0000 34'28 94:104 [1,0,2] 1 [1,0,2] 1 34'28 2024-09-15T20:36:52.759747+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 queued for scrub 0 0 2024-09-15T20:39:00.014 INFO:tasks.workunit.client.0.smithi139.stdout:1.1a 4 0 0 0 0 2468 0 0 18 12 18 active+clean 2024-09-15T20:38:53.430040+0000 44'30 94:134 [1,2,0] 1 [1,2,0] 1 44'30 2024-09-15T20:36:53.716793+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:21:15.320321+0000 3 0 2024-09-15T20:39:00.014 INFO:tasks.workunit.client.0.smithi139.stdout:1.1b 5 1 1 0 0 3085 0 0 19 12 19 active+recovery_wait+degraded 2024-09-15T20:38:53.430804+0000 45'31 94:151 [1,0,2] 1 [1,0,2] 1 41'30 2024-09-15T20:36:41.658323+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 no scrub is scheduled 4 0 2024-09-15T20:39:00.014 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:39:00.014 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:39:00.014 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:39:00.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-09-15T20:39:00.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-15T20:39:00.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:39:00.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:39:00.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:39:00.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-15T20:39:00.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:39:00.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:39:00.031 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:39:00.031 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:39:00.031 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T20:39:00.031 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T20:39:00.031 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:39:00.031 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:39:00.031 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:39:00.031 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:39:00.031 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:39:00.031 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:39:00.031 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:39:00.031 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:39:00.031 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:39:00.031 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:39:00.033 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:39:00.033 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:39:00.033 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:39:00.033 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:39:00.033 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:39:00.034 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:39:00.034 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:39:00.034 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:39:00.034 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:39:00.034 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:39:00.034 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-15T20:39:00.034 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:39:00.034 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.1 2024-09-15T20:39:00.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:39:00.035 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-15T20:39:00.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:39:00.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:39:00.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:39:00.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:39:00.067 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:39:00.066+0000 7f4c8bc8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:39:00.069 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:39:00.070+0000 7f4c8bc8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:39:00.071 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:39:00.070+0000 7f4c8bc8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:39:00.503 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:39:00.503 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:39:00.504 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:39:00.504 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:39:00.504 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:39:00.504 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:00.504 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:39:00.504 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:39:00.504 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:00.505 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:39:00.968 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 92 up_thru 92 down_at 67 last_clean_interval [10,64) [v2:127.0.0.1:6818/2815298135,v1:127.0.0.1:6819/2815298135] [v2:127.0.0.1:6820/2815298135,v1:127.0.0.1:6821/2815298135] exists,up d5b37d03-d9a9-4f3f-8c3e-d61cdbdce38e 2024-09-15T20:39:00.969 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:39:00.969 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:39:00.969 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:39:00.969 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: wait_for_osd up 1 2024-09-15T20:39:00.969 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:39:00.969 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:39:00.969 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:39:00.969 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:39:00.969 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:00.970 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:39:00.970 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:39:00.970 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:00.970 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:39:01.131 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:39:01.130+0000 7f4c8bc8a7c0 -1 Falling back to public interface 2024-09-15T20:39:01.460 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 92 up_thru 92 down_at 67 last_clean_interval [10,64) [v2:127.0.0.1:6818/2815298135,v1:127.0.0.1:6819/2815298135] [v2:127.0.0.1:6820/2815298135,v1:127.0.0.1:6821/2815298135] exists,up d5b37d03-d9a9-4f3f-8c3e-d61cdbdce38e 2024-09-15T20:39:01.461 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:39:01.461 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:39:01.461 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:39:01.461 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: sleep 20 2024-09-15T20:39:03.878 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:39:03.882+0000 7f4c8bc8a7c0 -1 osd.1 94 log_to_monitors true 2024-09-15T20:39:21.463 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: get_asok_path osd.1 2024-09-15T20:39:21.464 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T20:39:21.464 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T20:39:21.464 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:39:21.464 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:39:21.464 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:39:21.465 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.1.asok 2024-09-15T20:39:21.465 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: CEPH_ARGS= 2024-09-15T20:39:21.465 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T20:39:21.581 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:39:21.581 INFO:tasks.workunit.client.0.smithi139.stdout: "ops": [], 2024-09-15T20:39:21.581 INFO:tasks.workunit.client.0.smithi139.stdout: "num_ops": 0 2024-09-15T20:39:21.581 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:39:21.593 INFO:tasks.workunit.client.0.smithi139.stdout:allowing recovery 2024-09-15T20:39:21.594 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:809: TEST_divergent_3: echo 'allowing recovery' 2024-09-15T20:39:21.594 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: ceph pg dump pgs 2024-09-15T20:39:22.018 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:39:22.018 INFO:tasks.workunit.client.0.smithi139.stdout:1.1f 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:39:05.925015+0000 41'29 99:138 [2,0,1] 2 [2,0,1] 2 41'29 2024-09-15T20:37:07.497422+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:16:27.549078+0000 2 0 2024-09-15T20:39:22.018 INFO:tasks.workunit.client.0.smithi139.stdout:1.1e 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:39:05.916847+0000 46'29 99:164 [1,2,0] 1 [1,2,0] 1 40'28 2024-09-15T20:36:45.584734+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:00:56.751288+0000 2 0 2024-09-15T20:39:22.018 INFO:tasks.workunit.client.0.smithi139.stdout:1.1d 4 0 0 0 0 2468 0 0 19 12 19 active+clean 2024-09-15T20:39:05.919699+0000 42'31 99:149 [2,1,0] 2 [2,1,0] 2 36'29 2024-09-15T20:36:37.493943+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:37:27.926830+0000 2 0 2024-09-15T20:39:22.018 INFO:tasks.workunit.client.0.smithi139.stdout:1.b 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:39:05.921655+0000 45'29 99:152 [1,0,2] 1 [1,0,2] 1 34'28 2024-09-15T20:36:44.549410+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:03:36.852097+0000 2 0 2024-09-15T20:39:22.018 INFO:tasks.workunit.client.0.smithi139.stdout:1.a 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:39:05.917718+0000 39'29 99:122 [1,2,0] 1 [1,2,0] 1 39'29 2024-09-15T20:36:56.804751+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:17:58.793715+0000 3 0 2024-09-15T20:39:22.018 INFO:tasks.workunit.client.0.smithi139.stdout:1.9 5 1 1 0 0 3085 0 0 17 12 17 active+recovery_wait+degraded 2024-09-15T20:39:05.928228+0000 45'29 99:151 [1,2,0] 1 [1,2,0] 1 45'29 2024-09-15T20:36:51.656322+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 no scrub is scheduled 3 0 2024-09-15T20:39:22.018 INFO:tasks.workunit.client.0.smithi139.stdout:1.8 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-15T20:39:08.016333+0000 45'30 99:154 [1,2,0] 1 [1,2,0] 1 45'30 2024-09-15T20:39:08.016199+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T22:32:38.659466+0000 3 0 2024-09-15T20:39:22.018 INFO:tasks.workunit.client.0.smithi139.stdout:1.7 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-15T20:39:05.920147+0000 38'28 99:137 [1,2,0] 1 [1,2,0] 1 38'28 2024-09-15T20:38:54.554790+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:06:08.978447+0000 2 0 2024-09-15T20:39:22.019 INFO:tasks.workunit.client.0.smithi139.stdout:1.6 5 0 0 0 0 3085 0 0 19 12 19 active+clean 2024-09-15T20:39:05.916473+0000 94'31 99:134 [1,0,2] 1 [1,0,2] 1 42'30 2024-09-15T20:36:46.625811+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:50:35.155765+0000 2 0 2024-09-15T20:39:22.019 INFO:tasks.workunit.client.0.smithi139.stdout:1.5 6 0 0 0 0 3702 0 0 20 12 20 active+clean 2024-09-15T20:39:05.925715+0000 45'32 99:162 [2,0,1] 2 [2,0,1] 2 40'30 2024-09-15T20:36:40.468115+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T05:42:41.237246+0000 4 0 2024-09-15T20:39:22.019 INFO:tasks.workunit.client.0.smithi139.stdout:1.4 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-15T20:39:05.926539+0000 40'28 99:115 [1,0,2] 1 [1,0,2] 1 32'27 2024-09-15T20:36:38.561548+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:13:56.334830+0000 1 0 2024-09-15T20:39:22.019 INFO:tasks.workunit.client.0.smithi139.stdout:1.3 5 0 0 0 0 3085 0 0 18 12 18 active+clean 2024-09-15T20:39:06.017911+0000 47'30 99:149 [1,2,0] 1 [1,2,0] 1 47'30 2024-09-15T20:39:06.017793+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T05:58:24.618163+0000 5 0 2024-09-15T20:39:22.019 INFO:tasks.workunit.client.0.smithi139.stdout:1.2 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-15T20:39:05.917469+0000 42'28 99:133 [0,1,2] 0 [0,1,2] 0 42'28 2024-09-15T20:36:44.289898+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 periodic scrub scheduled @ 2024-09-17T07:45:48.713833+0000 1 0 2024-09-15T20:39:22.019 INFO:tasks.workunit.client.0.smithi139.stdout:1.c 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:39:05.919254+0000 44'29 99:127 [1,0,2] 1 [1,0,2] 1 38'28 2024-09-15T20:36:43.561129+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:23:05.183421+0000 2 0 2024-09-15T20:39:22.019 INFO:tasks.workunit.client.0.smithi139.stdout:1.1 2 0 0 0 0 1234 0 0 17 12 17 active+clean 2024-09-15T20:39:05.925959+0000 44'29 99:139 [2,0,1] 2 [2,0,1] 2 32'27 2024-09-15T20:36:42.549754+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T05:44:19.350450+0000 0 0 2024-09-15T20:39:22.019 INFO:tasks.workunit.client.0.smithi139.stdout:1.1c 2 0 0 0 0 1234 0 0 17 12 17 active+clean 2024-09-15T20:39:05.926289+0000 36'29 99:106 [1,0,2] 1 [1,0,2] 1 34'28 2024-09-15T20:36:34.684895+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:09:54.414496+0000 1 0 2024-09-15T20:39:22.019 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:39:05.912690+0000 45'29 99:130 [1,0,2] 1 [1,0,2] 1 32'27 2024-09-15T20:36:31.575682+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:58:36.674248+0000 1 0 2024-09-15T20:39:22.019 INFO:tasks.workunit.client.0.smithi139.stdout:1.d 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:39:05.926016+0000 45'29 99:148 [2,1,0] 2 [2,1,0] 2 32'27 2024-09-15T20:36:39.548519+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:32:10.808600+0000 1 0 2024-09-15T20:39:22.019 INFO:tasks.workunit.client.0.smithi139.stdout:1.e 3 1 1 0 0 2105 0 0 18 12 18 active+recovery_wait+degraded 2024-09-15T20:39:05.925942+0000 87'30 99:156 [1,2,0] 1 [1,2,0] 1 44'29 2024-09-15T20:36:49.578000+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 no scrub is scheduled 0 0 2024-09-15T20:39:22.020 INFO:tasks.workunit.client.0.smithi139.stdout:1.f 4 0 0 0 0 2468 0 0 17 12 17 active+clean 2024-09-15T20:39:05.914886+0000 41'29 99:138 [2,0,1] 2 [2,0,1] 2 41'29 2024-09-15T20:37:08.463113+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T22:13:29.100440+0000 3 0 2024-09-15T20:39:22.020 INFO:tasks.workunit.client.0.smithi139.stdout:1.10 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-15T20:39:05.923104+0000 45'30 99:154 [0,2,1] 0 [0,2,1] 0 38'29 2024-09-15T20:36:39.578100+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:33:49.685895+0000 2 0 2024-09-15T20:39:22.020 INFO:tasks.workunit.client.0.smithi139.stdout:1.11 3 0 1 0 0 1851 0 0 17 12 17 active+recovering+degraded 2024-09-15T20:39:05.930788+0000 37'29 99:124 [0,1,2] 0 [0,1,2] 0 36'28 2024-09-15T20:36:36.444590+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 no scrub is scheduled 2 0 2024-09-15T20:39:22.020 INFO:tasks.workunit.client.0.smithi139.stdout:1.12 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-15T20:39:05.927459+0000 36'28 99:139 [0,1,2] 0 [0,1,2] 0 32'27 2024-09-15T20:36:35.623615+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:45:05.820412+0000 0 0 2024-09-15T20:39:22.020 INFO:tasks.workunit.client.0.smithi139.stdout:1.13 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-15T20:39:09.059490+0000 45'30 99:149 [1,0,2] 1 [1,0,2] 1 45'30 2024-09-15T20:39:09.059363+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:43:16.011573+0000 3 0 2024-09-15T20:39:22.020 INFO:tasks.workunit.client.0.smithi139.stdout:1.14 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:39:05.796414+0000 45'29 99:149 [2,0,1] 2 [2,0,1] 2 32'27 2024-09-15T20:36:37.554032+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:37:35.265890+0000 1 0 2024-09-15T20:39:22.020 INFO:tasks.workunit.client.0.smithi139.stdout:1.15 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:39:05.912251+0000 41'29 99:122 [1,0,2] 1 [1,0,2] 1 41'29 2024-09-15T20:38:53.664450+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:24:46.677605+0000 3 0 2024-09-15T20:39:22.020 INFO:tasks.workunit.client.0.smithi139.stdout:1.16 4 0 0 0 0 2468 0 0 19 12 19 active+clean 2024-09-15T20:39:05.913995+0000 41'31 99:142 [1,0,2] 1 [1,0,2] 1 41'31 2024-09-15T20:36:47.576518+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:29:31.725575+0000 3 0 2024-09-15T20:39:22.020 INFO:tasks.workunit.client.0.smithi139.stdout:1.17 1 0 0 0 0 617 0 0 15 12 15 active+clean 2024-09-15T20:39:05.924639+0000 32'27 99:110 [0,1,2] 0 [0,1,2] 0 32'27 2024-09-15T20:38:53.869510+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T05:44:01.146938+0000 1 0 2024-09-15T20:39:22.020 INFO:tasks.workunit.client.0.smithi139.stdout:1.18 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-15T20:39:05.924219+0000 45'30 99:154 [2,1,0] 2 [2,1,0] 2 34'28 2024-09-15T20:36:36.622678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:23:01.045973+0000 1 0 2024-09-15T20:39:22.020 INFO:tasks.workunit.client.0.smithi139.stdout:1.19 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-15T20:39:06.989908+0000 34'28 99:123 [1,0,2] 1 [1,0,2] 1 34'28 2024-09-15T20:39:06.989781+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:43:55.173278+0000 2 0 2024-09-15T20:39:22.021 INFO:tasks.workunit.client.0.smithi139.stdout:1.1a 4 0 0 0 0 2468 0 0 18 12 18 active+clean 2024-09-15T20:39:05.920700+0000 44'30 99:145 [1,2,0] 1 [1,2,0] 1 44'30 2024-09-15T20:36:53.716793+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:09:02.399382+0000 3 0 2024-09-15T20:39:22.021 INFO:tasks.workunit.client.0.smithi139.stdout:1.1b 5 1 1 0 0 3085 0 0 19 12 19 active+recovery_wait+degraded 2024-09-15T20:39:05.917861+0000 45'31 99:160 [1,0,2] 1 [1,0,2] 1 41'30 2024-09-15T20:36:41.658323+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 no scrub is scheduled 4 0 2024-09-15T20:39:22.021 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:39:22.021 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:39:22.021 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:39:22.034 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-15T20:39:22.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-15T20:39:22.189 INFO:tasks.workunit.client.0.smithi139.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T20:39:22.203 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-15T20:39:22.203 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-15T20:39:22.352 INFO:tasks.workunit.client.0.smithi139.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T20:39:22.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-15T20:39:22.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-15T20:39:22.523 INFO:tasks.workunit.client.0.smithi139.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T20:39:22.537 INFO:tasks.workunit.client.0.smithi139.stdout:reading divergent objects 2024-09-15T20:39:22.537 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:817: TEST_divergent_3: echo 'reading divergent objects' 2024-09-15T20:39:22.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:818: TEST_divergent_3: ceph pg dump pgs 2024-09-15T20:39:22.950 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:39:22.951 INFO:tasks.workunit.client.0.smithi139.stdout:1.1f 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:39:05.925015+0000 41'29 99:138 [2,0,1] 2 [2,0,1] 2 41'29 2024-09-15T20:37:07.497422+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:16:27.549078+0000 2 0 2024-09-15T20:39:22.951 INFO:tasks.workunit.client.0.smithi139.stdout:1.1e 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:39:05.916847+0000 46'29 99:164 [1,2,0] 1 [1,2,0] 1 40'28 2024-09-15T20:36:45.584734+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:00:56.751288+0000 2 0 2024-09-15T20:39:22.951 INFO:tasks.workunit.client.0.smithi139.stdout:1.1d 4 0 0 0 0 2468 0 0 19 12 19 active+clean 2024-09-15T20:39:05.919699+0000 42'31 99:149 [2,1,0] 2 [2,1,0] 2 36'29 2024-09-15T20:36:37.493943+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:37:27.926830+0000 2 0 2024-09-15T20:39:22.951 INFO:tasks.workunit.client.0.smithi139.stdout:1.b 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:39:05.921655+0000 45'29 99:152 [1,0,2] 1 [1,0,2] 1 34'28 2024-09-15T20:36:44.549410+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:03:36.852097+0000 2 0 2024-09-15T20:39:22.951 INFO:tasks.workunit.client.0.smithi139.stdout:1.a 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:39:05.917718+0000 39'29 99:122 [1,2,0] 1 [1,2,0] 1 39'29 2024-09-15T20:36:56.804751+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:17:58.793715+0000 3 0 2024-09-15T20:39:22.951 INFO:tasks.workunit.client.0.smithi139.stdout:1.9 5 1 1 0 0 3085 0 0 17 12 17 active+recovery_wait+degraded 2024-09-15T20:39:05.928228+0000 45'29 99:151 [1,2,0] 1 [1,2,0] 1 45'29 2024-09-15T20:36:51.656322+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 no scrub is scheduled 3 0 2024-09-15T20:39:22.951 INFO:tasks.workunit.client.0.smithi139.stdout:1.8 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-15T20:39:08.016333+0000 45'30 99:154 [1,2,0] 1 [1,2,0] 1 45'30 2024-09-15T20:39:08.016199+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T22:32:38.659466+0000 3 0 2024-09-15T20:39:22.951 INFO:tasks.workunit.client.0.smithi139.stdout:1.7 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-15T20:39:05.920147+0000 38'28 99:137 [1,2,0] 1 [1,2,0] 1 38'28 2024-09-15T20:38:54.554790+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:06:08.978447+0000 2 0 2024-09-15T20:39:22.951 INFO:tasks.workunit.client.0.smithi139.stdout:1.6 5 0 0 0 0 3085 0 0 19 12 19 active+clean 2024-09-15T20:39:05.916473+0000 94'31 99:134 [1,0,2] 1 [1,0,2] 1 42'30 2024-09-15T20:36:46.625811+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T21:50:35.155765+0000 2 0 2024-09-15T20:39:22.951 INFO:tasks.workunit.client.0.smithi139.stdout:1.5 6 0 0 0 0 3702 0 0 20 12 20 active+clean 2024-09-15T20:39:05.925715+0000 45'32 99:162 [2,0,1] 2 [2,0,1] 2 40'30 2024-09-15T20:36:40.468115+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T05:42:41.237246+0000 4 0 2024-09-15T20:39:22.951 INFO:tasks.workunit.client.0.smithi139.stdout:1.4 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-15T20:39:05.926539+0000 40'28 99:115 [1,0,2] 1 [1,0,2] 1 32'27 2024-09-15T20:36:38.561548+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:13:56.334830+0000 1 0 2024-09-15T20:39:22.951 INFO:tasks.workunit.client.0.smithi139.stdout:1.3 5 0 0 0 0 3085 0 0 18 12 18 active+clean 2024-09-15T20:39:06.017911+0000 47'30 99:149 [1,2,0] 1 [1,2,0] 1 47'30 2024-09-15T20:39:06.017793+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T05:58:24.618163+0000 5 0 2024-09-15T20:39:22.952 INFO:tasks.workunit.client.0.smithi139.stdout:1.2 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-15T20:39:05.917469+0000 42'28 99:133 [0,1,2] 0 [0,1,2] 0 42'28 2024-09-15T20:36:44.289898+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 periodic scrub scheduled @ 2024-09-17T07:45:48.713833+0000 1 0 2024-09-15T20:39:22.952 INFO:tasks.workunit.client.0.smithi139.stdout:1.c 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:39:05.919254+0000 44'29 99:127 [1,0,2] 1 [1,0,2] 1 38'28 2024-09-15T20:36:43.561129+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T02:23:05.183421+0000 2 0 2024-09-15T20:39:22.952 INFO:tasks.workunit.client.0.smithi139.stdout:1.1 2 0 0 0 0 1234 0 0 17 12 17 active+clean 2024-09-15T20:39:05.925959+0000 44'29 99:139 [2,0,1] 2 [2,0,1] 2 32'27 2024-09-15T20:36:42.549754+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T05:44:19.350450+0000 0 0 2024-09-15T20:39:22.952 INFO:tasks.workunit.client.0.smithi139.stdout:1.1c 2 0 0 0 0 1234 0 0 17 12 17 active+clean 2024-09-15T20:39:05.926289+0000 36'29 99:106 [1,0,2] 1 [1,0,2] 1 34'28 2024-09-15T20:36:34.684895+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T08:09:54.414496+0000 1 0 2024-09-15T20:39:22.952 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:39:05.912690+0000 45'29 99:130 [1,0,2] 1 [1,0,2] 1 32'27 2024-09-15T20:36:31.575682+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:58:36.674248+0000 1 0 2024-09-15T20:39:22.952 INFO:tasks.workunit.client.0.smithi139.stdout:1.d 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:39:05.926016+0000 45'29 99:148 [2,1,0] 2 [2,1,0] 2 32'27 2024-09-15T20:36:39.548519+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T06:32:10.808600+0000 1 0 2024-09-15T20:39:22.952 INFO:tasks.workunit.client.0.smithi139.stdout:1.e 3 1 1 0 0 2105 0 0 18 12 18 active+recovery_wait+degraded 2024-09-15T20:39:05.925942+0000 87'30 99:156 [1,2,0] 1 [1,2,0] 1 44'29 2024-09-15T20:36:49.578000+0000 0'0 2024-09-15T20:36:11.616735+0000 0 0 no scrub is scheduled 0 0 2024-09-15T20:39:22.952 INFO:tasks.workunit.client.0.smithi139.stdout:1.f 4 0 0 0 0 2468 0 0 17 12 17 active+clean 2024-09-15T20:39:05.914886+0000 41'29 99:138 [2,0,1] 2 [2,0,1] 2 41'29 2024-09-15T20:37:08.463113+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T22:13:29.100440+0000 3 0 2024-09-15T20:39:22.952 INFO:tasks.workunit.client.0.smithi139.stdout:1.10 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-15T20:39:05.923104+0000 45'30 99:154 [0,2,1] 0 [0,2,1] 0 38'29 2024-09-15T20:36:39.578100+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T00:33:49.685895+0000 2 0 2024-09-15T20:39:22.952 INFO:tasks.workunit.client.0.smithi139.stdout:1.11 3 0 1 0 0 1851 0 0 17 12 17 active+recovering+degraded 2024-09-15T20:39:05.930788+0000 37'29 99:124 [0,1,2] 0 [0,1,2] 0 36'28 2024-09-15T20:36:36.444590+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 no scrub is scheduled 2 0 2024-09-15T20:39:22.952 INFO:tasks.workunit.client.0.smithi139.stdout:1.12 1 0 0 0 0 617 0 0 16 12 16 active+clean 2024-09-15T20:39:05.927459+0000 36'28 99:139 [0,1,2] 0 [0,1,2] 0 32'27 2024-09-15T20:36:35.623615+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:45:05.820412+0000 0 0 2024-09-15T20:39:22.952 INFO:tasks.workunit.client.0.smithi139.stdout:1.13 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-15T20:39:09.059490+0000 45'30 99:149 [1,0,2] 1 [1,0,2] 1 45'30 2024-09-15T20:39:09.059363+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:43:16.011573+0000 3 0 2024-09-15T20:39:22.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.14 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:39:05.796414+0000 45'29 99:149 [2,0,1] 2 [2,0,1] 2 32'27 2024-09-15T20:36:37.554032+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-16T20:37:35.265890+0000 1 0 2024-09-15T20:39:22.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.15 3 0 0 0 0 1851 0 0 17 12 17 active+clean 2024-09-15T20:39:05.912251+0000 41'29 99:122 [1,0,2] 1 [1,0,2] 1 41'29 2024-09-15T20:38:53.664450+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T07:24:46.677605+0000 3 0 2024-09-15T20:39:22.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.16 4 0 0 0 0 2468 0 0 19 12 19 active+clean 2024-09-15T20:39:05.913995+0000 41'31 99:142 [1,0,2] 1 [1,0,2] 1 41'31 2024-09-15T20:36:47.576518+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:29:31.725575+0000 3 0 2024-09-15T20:39:22.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.17 1 0 0 0 0 617 0 0 15 12 15 active+clean 2024-09-15T20:39:05.924639+0000 32'27 99:110 [0,1,2] 0 [0,1,2] 0 32'27 2024-09-15T20:38:53.869510+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T05:44:01.146938+0000 1 0 2024-09-15T20:39:22.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.18 3 0 0 0 0 1851 0 0 18 12 18 active+clean 2024-09-15T20:39:05.924219+0000 45'30 99:154 [2,1,0] 2 [2,1,0] 2 34'28 2024-09-15T20:36:36.622678+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T01:23:01.045973+0000 1 0 2024-09-15T20:39:22.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.19 2 0 0 0 0 1234 0 0 16 12 16 active+clean 2024-09-15T20:39:06.989908+0000 34'28 99:123 [1,0,2] 1 [1,0,2] 1 34'28 2024-09-15T20:39:06.989781+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T04:43:55.173278+0000 2 0 2024-09-15T20:39:22.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.1a 4 0 0 0 0 2468 0 0 18 12 18 active+clean 2024-09-15T20:39:05.920700+0000 44'30 99:145 [1,2,0] 1 [1,2,0] 1 44'30 2024-09-15T20:36:53.716793+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 periodic scrub scheduled @ 2024-09-17T03:09:02.399382+0000 3 0 2024-09-15T20:39:22.953 INFO:tasks.workunit.client.0.smithi139.stdout:1.1b 5 1 1 0 0 3085 0 0 19 12 19 active+recovery_wait+degraded 2024-09-15T20:39:05.917861+0000 45'31 99:160 [1,0,2] 1 [1,0,2] 1 41'30 2024-09-15T20:36:41.658323+0000 0'0 2024-09-15T20:36:11.616735+0000 0 1 no scrub is scheduled 4 0 2024-09-15T20:39:22.953 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:39:22.953 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:39:22.953 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:39:22.969 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: expr 2 + 2 2024-09-15T20:39:22.970 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: seq 1 4 2024-09-15T20:39:22.971 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T20:39:22.971 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2024-09-15T20:39:23.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T20:39:23.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2024-09-15T20:39:23.050 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T20:39:23.050 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2024-09-15T20:39:23.083 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T20:39:23.083 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2024-09-15T20:39:23.117 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: seq 1 2 2024-09-15T20:39:23.118 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-15T20:39:23.118 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2024-09-15T20:39:23.150 INFO:tasks.workunit.client.0.smithi139.stderr:error getting test/newobject_1: (2) No such file or directory 2024-09-15T20:39:23.152 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-15T20:39:23.152 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2024-09-15T20:39:23.185 INFO:tasks.workunit.client.0.smithi139.stderr:error getting test/newobject_2: (2) No such file or directory 2024-09-15T20:39:23.188 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: rm -f td/divergent-priors/existing 2024-09-15T20:39:23.188 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-15T20:39:23.189 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-15T20:39:23.211 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:52.958+0000 7f0d786f5640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 54'30 (31'2) modify 1:97db2a0f:::existing_2:head by client.4650.0:1 2024-09-15T20:37:01.184540+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-15T20:39:23.212 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:52.958+0000 7f0d786f5640 20 _merge_object_divergent_entries: keeping 54'30 (31'2) modify 1:97db2a0f:::existing_2:head by client.4650.0:1 2024-09-15T20:37:01.184540+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-15T20:39:23.212 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:52.958+0000 7f0d786f5640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T20:39:23.212 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:52.958+0000 7f0d786f5640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 31'2 first_divergent_update: 54'30 last_divergent_update: 54'30 2024-09-15T20:39:23.212 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:52.958+0000 7f0d786f5640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-09-15T20:39:23.212 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:52.958+0000 7f0d786f5640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-09-15T20:39:23.213 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:52.958+0000 7f0d786f5640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 54'30 2024-09-15T20:39:23.213 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:52.958+0000 7f0d786f5640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 54'30 (31'2) modify 1:97db2a0f:::existing_2:head by client.4650.0:1 2024-09-15T20:37:01.184540+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-15T20:39:23.213 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:52.958+0000 7f0d786f5640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-09-15T20:39:23.213 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:52.958+0000 7f0d786f5640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 31'2 <= info.log_tail 31'12 2024-09-15T20:39:23.213 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d796f7640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 54'33 (31'1) modify 1:d858c035:::existing_1:head by client.4643.0:1 2024-09-15T20:37:01.170545+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-15T20:39:23.213 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d796f7640 20 _merge_object_divergent_entries: keeping 54'33 (31'1) modify 1:d858c035:::existing_1:head by client.4643.0:1 2024-09-15T20:37:01.170545+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-15T20:39:23.213 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T20:39:23.213 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 31'1 first_divergent_update: 54'33 last_divergent_update: 54'33 2024-09-15T20:39:23.213 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-09-15T20:39:23.213 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-09-15T20:39:23.213 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 54'33 2024-09-15T20:39:23.213 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 54'33 (31'1) modify 1:d858c035:::existing_1:head by client.4643.0:1 2024-09-15T20:37:01.170545+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-15T20:39:23.214 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-09-15T20:39:23.214 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 31'1 <= info.log_tail 31'12 2024-09-15T20:39:23.214 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d796f7640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 54'32 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4640.0:1 2024-09-15T20:37:01.165629+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-15T20:39:23.214 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d796f7640 20 _merge_object_divergent_entries: keeping 54'32 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4640.0:1 2024-09-15T20:37:01.165629+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-15T20:39:23.214 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T20:39:23.214 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 54'32 last_divergent_update: 54'32 2024-09-15T20:39:23.214 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-09-15T20:39:23.214 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-09-15T20:39:23.214 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d786f5640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 54'31 (31'4) delete 1:70b3bf12:::existing_4:head by client.4653.0:1 2024-09-15T20:37:01.188794+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-15T20:39:23.214 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d786f5640 20 _merge_object_divergent_entries: keeping 54'31 (31'4) delete 1:70b3bf12:::existing_4:head by client.4653.0:1 2024-09-15T20:37:01.188794+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-15T20:39:23.214 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d786f5640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T20:39:23.215 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d786f5640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 31'4 first_divergent_update: 54'31 last_divergent_update: 54'31 2024-09-15T20:39:23.215 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d786f5640 10 _merge_object_divergent_entries: more recent entry found: 87'30 (31'4) modify 1:70b3bf12:::existing_4:head by client.4772.0:1 2024-09-15T20:38:06.763694+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false, already merged 2024-09-15T20:39:23.215 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d786f5640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 54'30 (0'0) modify 1:75ac5091:::newobject_2:head by client.4646.0:1 2024-09-15T20:37:01.177371+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-15T20:39:23.215 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d786f5640 20 _merge_object_divergent_entries: keeping 54'30 (0'0) modify 1:75ac5091:::newobject_2:head by client.4646.0:1 2024-09-15T20:37:01.177371+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-15T20:39:23.224 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d786f5640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T20:39:23.224 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d786f5640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 54'30 last_divergent_update: 54'30 2024-09-15T20:39:23.225 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d786f5640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-09-15T20:39:23.225 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.050+0000 7f0d786f5640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-09-15T20:39:23.225 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.566+0000 7f0d796f7640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 54'30 (31'3) delete 1:8881e6d8:::existing_3:head by client.4655.0:1 2024-09-15T20:37:01.191024+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-15T20:39:23.225 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.566+0000 7f0d796f7640 20 _merge_object_divergent_entries: keeping 54'30 (31'3) delete 1:8881e6d8:::existing_3:head by client.4655.0:1 2024-09-15T20:37:01.191024+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-15T20:39:23.225 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.566+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-15T20:39:23.225 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.566+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 54'30 last_divergent_update: 54'30 2024-09-15T20:39:23.226 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.566+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-15T20:39:23.226 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.566+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-15T20:39:23.226 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.566+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 54'30 2024-09-15T20:39:23.226 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.566+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 54'30 (31'3) delete 1:8881e6d8:::existing_3:head by client.4655.0:1 2024-09-15T20:37:01.191024+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-15T20:39:23.226 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.566+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-15T20:39:23.226 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:38:53.566+0000 7f0d796f7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 31'12 2024-09-15T20:39:23.227 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-15T20:39:23.227 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-15T20:39:23.248 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-15T20:39:23.249 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-15T20:39:23.269 INFO:tasks.workunit.client.0.smithi139.stdout:success 2024-09-15T20:39:23.269 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:842: TEST_divergent_3: echo success 2024-09-15T20:39:23.269 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:844: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.27419.out 2024-09-15T20:39:23.270 INFO:tasks.workunit.client.0.smithi139.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.27419.out': No such file or directory 2024-09-15T20:39:23.270 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:846: TEST_divergent_3: delete_pool test 2024-09-15T20:39:23.270 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-15T20:39:23.270 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-15T20:39:23.821 INFO:tasks.workunit.client.0.smithi139.stderr:pool 'test' does not exist 2024-09-15T20:39:23.839 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:847: TEST_divergent_3: kill_daemons td/divergent-priors 2024-09-15T20:39:23.840 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:39:23.840 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:39:23.840 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:39:23.841 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:39:23.841 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:39:24.151 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:39:24.152 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-15T20:39:24.152 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-15T20:39:24.152 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:39:24.152 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-15T20:39:24.152 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:39:24.153 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:39:24.153 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:39:24.153 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:39:24.153 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:39:24.159 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:39:24.159 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:39:24.160 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:39:24.160 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:39:24.161 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:39:24.161 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:39:24.162 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:39:24.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:39:24.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:39:24.163 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:39:24.163 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:39:24.163 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:39:24.164 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:39:24.165 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-15T20:39:24.198 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:39:24.198 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:39:24.198 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:39:24.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27419 2024-09-15T20:39:24.199 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:39:24.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:39:24.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-15T20:39:24.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-15T20:39:24.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-15T20:39:24.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-15T20:39:24.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-15T20:39:24.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:39:24.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-15T20:39:24.200 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:39:24.200 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:39:24.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:39:24.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:39:24.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:39:24.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:39:24.202 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:39:24.202 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:39:24.203 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:39:24.203 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:39:24.204 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:39:24.204 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:39:24.204 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:39:24.205 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:39:24.205 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:39:24.205 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:39:24.206 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:39:24.207 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:39:24.207 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-15T20:39:24.208 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:39:24.208 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:39:24.208 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:39:24.208 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27419 2024-09-15T20:39:24.209 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:39:24.209 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:39:24.209 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-15T20:39:24.213 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:39:24.213 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:39:24.213 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:39:24.213 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27419 2024-09-15T20:39:24.213 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:39:24.213 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T20:39:24.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:39:24.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-15T20:39:24.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2024-09-15T20:39:24.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2024-09-15T20:39:24.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2024-09-15T20:39:24.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2024-09-15T20:39:24.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2024-09-15T20:39:24.214 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2024-09-15T20:39:24.214 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2024-09-15T20:39:24.215 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2024-09-15T20:39:24.215 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:39:24.215 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:39:24.215 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2024-09-15T20:39:24.215 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-15T20:39:24.215 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T20:39:24.215 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T20:39:24.216 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T20:39:24.216 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-15T20:39:24.216 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-15T20:39:24.676 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T20:39:24.676 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:39:24.676 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:39:24.677 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:39:24.677 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:39:24.677 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:39:24.677 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:39:24.678 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T20:39:24.904 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T20:39:24.905 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T20:39:24.905 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:39:24.906 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:39:24.906 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T20:39:24.906 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T20:39:24.907 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:39:24.907 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:39:24.907 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:39:24.907 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:39:24.907 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:39:24.907 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:39:24.907 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-mon.a.asok 2024-09-15T20:39:24.907 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:39:24.908 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27419/ceph-mon.a.asok config get fsid 2024-09-15T20:39:25.018 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T20:39:25.018 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:39:25.018 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:39:25.018 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T20:39:25.019 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T20:39:25.019 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:39:25.019 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:39:25.019 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:39:25.020 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:39:25.020 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:39:25.020 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:39:25.020 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-mon.a.asok 2024-09-15T20:39:25.020 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:39:25.020 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27419/ceph-mon.a.asok config get mon_host 2024-09-15T20:39:25.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2024-09-15T20:39:25.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-15T20:39:25.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T20:39:25.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T20:39:25.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T20:39:25.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-15T20:39:25.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T20:39:26.042 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T20:39:26.042 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:39:26.042 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:39:26.043 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:39:26.043 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:39:26.043 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:39:26.043 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:39:26.044 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:39:26.046 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:39:26.119 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-15T20:39:26.119 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2024-09-15T20:39:26.119 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-15T20:39:26.119 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:39:26.119 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T20:39:26.119 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:39:26.119 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-15T20:39:26.119 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:39:26.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:39:26.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:39:26.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:39:26.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-15T20:39:26.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-15T20:39:26.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:39:26.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:39:26.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:39:26.120 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:39:26.120 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:39:26.120 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:39:26.122 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:39:26.122 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:39:26.122 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:39:26.122 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:39:26.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:39:26.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:39:26.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:39:26.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:39:26.123 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:39:26.123 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:39:26.123 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:39:26.123 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:39:26.123 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:39:26.123 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:39:26.123 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:39:26.123 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:39:26.123 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-15T20:39:26.123 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:39:26.124 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=072ef209-0424-4dcb-ba88-59a19eaa46bd 2024-09-15T20:39:26.124 INFO:tasks.workunit.client.0.smithi139.stdout:add osd0 072ef209-0424-4dcb-ba88-59a19eaa46bd 2024-09-15T20:39:26.124 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 072ef209-0424-4dcb-ba88-59a19eaa46bd' 2024-09-15T20:39:26.124 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:39:26.148 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD+RedmzZ8LCRAAQ0EnmMe/xLLv6Fxf68wzig== 2024-09-15T20:39:26.148 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD+RedmzZ8LCRAAQ0EnmMe/xLLv6Fxf68wzig=="}' 2024-09-15T20:39:26.148 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 072ef209-0424-4dcb-ba88-59a19eaa46bd -i td/divergent-priors/0/new.json 2024-09-15T20:39:26.474 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:39:26.490 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-15T20:39:26.491 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+RedmzZ8LCRAAQ0EnmMe/xLLv6Fxf68wzig== --osd-uuid 072ef209-0424-4dcb-ba88-59a19eaa46bd 2024-09-15T20:39:26.540 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:39:26.542+0000 7f594e7d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:39:26.550 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:39:26.554+0000 7f594e7d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:39:26.552 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:39:26.554+0000 7f594e7d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:39:26.552 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:39:26.554+0000 7f594e7d17c0 -1 bdev(0x555b6452bc00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:39:26.552 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:39:26.554+0000 7f594e7d17c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-15T20:39:32.735 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-15T20:39:32.736 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:39:32.737 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T20:39:32.737 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd0 key to auth repository 2024-09-15T20:39:32.738 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:39:33.332 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T20:39:33.333 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T20:39:33.333 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:39:33.333 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:39:33.333 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:39:33.334 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:39:33.372 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:39:33.373+0000 7f6d5f9c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:39:33.405 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:39:33.405+0000 7f6d5f9c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:39:33.407 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:39:33.409+0000 7f6d5f9c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:39:33.844 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T20:39:33.844 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:39:33.844 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:39:33.844 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:39:33.844 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:39:33.844 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:33.844 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:39:33.844 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:39:33.845 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:33.845 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:34.337 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:34.652 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:39:34.653+0000 7f6d5f9c47c0 -1 Falling back to public interface 2024-09-15T20:39:35.338 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:35.338 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:35.338 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:39:35.339 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:39:35.339 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:35.339 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:35.781 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:36.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:36.783 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:39:36.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:36.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:39:36.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:36.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:37.246 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:37.504 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:39:37.505+0000 7f6d5f9c47c0 -1 osd.0 0 log_to_monitors true 2024-09-15T20:39:38.247 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:38.248 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:38.248 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:39:38.248 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:39:38.248 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:38.248 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:38.768 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:39.770 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:39.770 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:39.770 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:39:39.771 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:39:39.771 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:39.771 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:40.388 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:41.390 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:41.390 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:41.390 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:39:41.391 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:39:41.391 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:41.391 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:41.861 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:42.863 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:42.863 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:42.863 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:39:42.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:39:42.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:42.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:43.365 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:44.366 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:44.366 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:44.367 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:39:44.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:39:44.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:44.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:44.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:45.812 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:45.812 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:45.813 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:39:45.813 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:39:45.813 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:45.813 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:46.431 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:47.432 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:47.432 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:47.433 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:39:47.433 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:39:47.433 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:47.433 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:47.895 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:48.897 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:48.897 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:48.898 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:39:48.898 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:39:48.898 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:48.898 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:49.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:50.459 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:50.459 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:50.459 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:39:50.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:39:50.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:50.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:51.598 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:52.599 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:52.600 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:52.600 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:39:52.600 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:39:52.600 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:52.601 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:53.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:54.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:54.123 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:54.123 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:39:54.123 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:39:54.123 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:54.124 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:54.896 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:55.898 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:55.898 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:55.899 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:39:55.899 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:39:55.899 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:55.899 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:56.700 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:57.702 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:57.702 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:57.703 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:39:57.703 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:39:57.703 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:57.703 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:39:58.915 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:39:59.917 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:39:59.917 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:39:59.917 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:39:59.918 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:39:59.918 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:39:59.918 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:40:00.828 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:01.830 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:01.831 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:01.831 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:40:01.831 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:40:01.831 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:01.832 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:40:02.309 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3978700634,v1:127.0.0.1:6803/3978700634] [v2:127.0.0.1:6804/3978700634,v1:127.0.0.1:6805/3978700634] exists,up 072ef209-0424-4dcb-ba88-59a19eaa46bd 2024-09-15T20:40:02.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:40:02.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:40:02.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:40:02.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-15T20:40:02.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2024-09-15T20:40:02.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-15T20:40:02.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:40:02.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T20:40:02.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:40:02.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-15T20:40:02.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:40:02.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:40:02.312 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:40:02.312 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:40:02.312 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T20:40:02.312 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T20:40:02.312 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:40:02.312 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:40:02.312 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:40:02.312 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:40:02.312 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:40:02.312 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:40:02.314 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:40:02.314 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:40:02.314 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:40:02.314 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:40:02.314 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:40:02.314 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:40:02.314 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:40:02.314 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:40:02.315 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:40:02.315 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:40:02.315 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:40:02.315 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:40:02.315 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:40:02.315 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:40:02.315 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:40:02.315 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:40:02.315 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-15T20:40:02.316 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:40:02.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=61477bed-b55a-47e6-8ac3-ac4772590dce 2024-09-15T20:40:02.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 61477bed-b55a-47e6-8ac3-ac4772590dce' 2024-09-15T20:40:02.316 INFO:tasks.workunit.client.0.smithi139.stdout:add osd1 61477bed-b55a-47e6-8ac3-ac4772590dce 2024-09-15T20:40:02.317 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:40:02.344 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAiRudmi8K6FBAAHYrl6a3ioK3PJSSbaZb7ng== 2024-09-15T20:40:02.344 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAiRudmi8K6FBAAHYrl6a3ioK3PJSSbaZb7ng=="}' 2024-09-15T20:40:02.344 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 61477bed-b55a-47e6-8ac3-ac4772590dce -i td/divergent-priors/1/new.json 2024-09-15T20:40:02.911 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:40:02.928 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-15T20:40:02.929 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAiRudmi8K6FBAAHYrl6a3ioK3PJSSbaZb7ng== --osd-uuid 61477bed-b55a-47e6-8ac3-ac4772590dce 2024-09-15T20:40:02.971 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:02.973+0000 7f1ae5fff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:03.025 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:03.029+0000 7f1ae5fff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:03.027 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:03.029+0000 7f1ae5fff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:03.027 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:03.029+0000 7f1ae5fff7c0 -1 bdev(0x55cdd9931c00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:40:03.027 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:03.029+0000 7f1ae5fff7c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-15T20:40:09.536 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-15T20:40:09.536 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:40:09.537 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd1 key to auth repository 2024-09-15T20:40:09.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T20:40:09.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:40:10.121 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.1 2024-09-15T20:40:10.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T20:40:10.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:40:10.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:40:10.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:40:10.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:40:10.161 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:10.165+0000 7f16bf14f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:10.188 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:10.189+0000 7f16bf14f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:10.189 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:10.189+0000 7f16bf14f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:10.576 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T20:40:10.576 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:40:10.577 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:40:10.577 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:40:10.577 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:40:10.577 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:40:10.577 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:10.577 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:40:10.578 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:10.578 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:11.026 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:11.416 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:11.417+0000 7f16bf14f7c0 -1 Falling back to public interface 2024-09-15T20:40:12.028 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:12.028 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:12.029 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:40:12.029 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:40:12.029 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:12.029 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:12.451 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:13.453 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:13.453 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:13.454 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:40:13.454 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:40:13.454 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:13.454 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:13.883 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:14.356 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:14.357+0000 7f16bf14f7c0 -1 osd.1 0 log_to_monitors true 2024-09-15T20:40:14.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:14.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:14.885 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:40:14.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:40:14.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:14.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:15.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:16.562 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:16.563 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:16.563 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:40:16.563 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:40:16.563 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:16.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:17.091 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:18.093 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:18.093 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:18.094 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:40:18.094 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:40:18.094 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:18.094 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:18.592 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:19.594 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:19.594 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:19.594 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:40:19.595 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:40:19.595 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:19.595 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:20.118 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:21.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:21.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:21.120 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:40:21.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:40:21.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:21.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:21.837 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:22.838 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:22.839 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:22.839 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:40:22.839 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:40:22.839 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:22.840 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:23.453 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:24.454 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:24.455 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:24.455 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:40:24.455 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:40:24.455 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:24.456 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:24.888 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:25.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:25.890 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:25.890 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:40:25.890 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:40:25.890 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:25.891 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:26.387 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:27.389 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:27.389 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:27.389 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:40:27.389 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:40:27.390 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:27.390 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:27.974 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:28.976 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:28.976 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:28.976 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:40:28.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:40:28.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:28.977 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:29.738 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:30.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:30.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:30.740 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:40:30.741 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:40:30.741 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:30.741 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:31.421 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:32.423 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:32.423 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:32.423 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:40:32.424 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:40:32.424 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:32.424 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:33.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:34.281 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:34.281 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:34.282 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:40:34.282 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:40:34.282 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:34.282 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:35.058 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:36.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:36.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:36.060 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:40:36.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:40:36.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:36.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:36.833 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:37.835 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:37.835 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:37.835 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:40:37.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:40:37.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:37.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:38.690 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:39.692 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:39.692 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:39.693 INFO:tasks.workunit.client.0.smithi139.stdout:18 2024-09-15T20:40:39.694 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:40:39.694 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:39.694 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:40.383 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:41.385 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:41.385 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:41.385 INFO:tasks.workunit.client.0.smithi139.stdout:19 2024-09-15T20:40:41.386 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:40:41.386 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:41.386 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:40:41.828 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3833330082,v1:127.0.0.1:6811/3833330082] [v2:127.0.0.1:6812/3833330082,v1:127.0.0.1:6813/3833330082] exists,up 61477bed-b55a-47e6-8ac3-ac4772590dce 2024-09-15T20:40:41.831 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:40:41.831 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:40:41.831 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:40:41.832 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-15T20:40:41.832 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2024-09-15T20:40:41.832 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-15T20:40:41.832 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:40:41.832 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T20:40:41.832 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:40:41.832 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-15T20:40:41.832 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:40:41.832 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:40:41.832 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:40:41.832 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:40:41.832 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-15T20:40:41.833 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-15T20:40:41.833 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:40:41.833 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:40:41.833 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:40:41.833 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:40:41.833 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:40:41.833 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:40:41.833 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:40:41.833 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:40:41.833 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:40:41.833 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:40:41.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:40:41.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:40:41.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:40:41.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:40:41.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:40:41.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:40:41.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:40:41.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:40:41.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:40:41.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:40:41.836 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:40:41.837 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:40:41.837 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-15T20:40:41.837 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:40:41.837 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e0a289bd-3c1f-48e6-b612-9a2c75ec33bb 2024-09-15T20:40:41.837 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 e0a289bd-3c1f-48e6-b612-9a2c75ec33bb' 2024-09-15T20:40:41.837 INFO:tasks.workunit.client.0.smithi139.stdout:add osd2 e0a289bd-3c1f-48e6-b612-9a2c75ec33bb 2024-09-15T20:40:41.837 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:40:41.863 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBJRudmWlqnMxAAvo+ChrApM4dox843yHO4Ow== 2024-09-15T20:40:41.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBJRudmWlqnMxAAvo+ChrApM4dox843yHO4Ow=="}' 2024-09-15T20:40:41.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e0a289bd-3c1f-48e6-b612-9a2c75ec33bb -i td/divergent-priors/2/new.json 2024-09-15T20:40:42.444 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:40:42.461 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-15T20:40:42.463 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBJRudmWlqnMxAAvo+ChrApM4dox843yHO4Ow== --osd-uuid e0a289bd-3c1f-48e6-b612-9a2c75ec33bb 2024-09-15T20:40:42.500 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:42.500+0000 7f90837c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:42.502 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:42.504+0000 7f90837c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:42.504 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:42.504+0000 7f90837c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:42.504 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:42.504+0000 7f90837c77c0 -1 bdev(0x560cecb8fc00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-15T20:40:42.505 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:42.504+0000 7f90837c77c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-15T20:40:48.926 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-15T20:40:48.926 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:40:48.928 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd2 key to auth repository 2024-09-15T20:40:48.928 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T20:40:48.928 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:40:49.521 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.2 2024-09-15T20:40:49.521 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T20:40:49.522 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:40:49.522 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:40:49.522 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:40:49.522 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:40:49.563 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:49.564+0000 7f716cd157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:49.586 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:49.588+0000 7f716cd157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:49.587 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:49.588+0000 7f716cd157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:40:50.016 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T20:40:50.016 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:40:50.016 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:40:50.016 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:40:50.016 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:40:50.016 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:50.016 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:40:50.017 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:40:50.017 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:50.017 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:40:50.490 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:50.868 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:50.868+0000 7f716cd157c0 -1 Falling back to public interface 2024-09-15T20:40:51.492 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:51.492 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:51.492 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:40:51.492 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:40:51.493 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:51.493 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:40:51.984 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:52.986 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:52.986 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:52.986 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:40:52.986 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:40:52.987 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:52.987 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:40:53.542 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:53.934 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:40:53.936+0000 7f716cd157c0 -1 osd.2 0 log_to_monitors true 2024-09-15T20:40:54.543 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:54.543 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:54.544 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:40:54.544 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:40:54.544 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:54.544 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:40:55.144 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:56.146 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:56.146 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:56.146 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:40:56.146 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:40:56.146 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:56.146 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:40:56.605 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:57.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:57.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:57.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:40:57.607 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:40:57.607 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:57.607 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:40:58.093 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:40:59.095 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:40:59.095 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:40:59.095 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:40:59.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:40:59.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:40:59.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:40:59.623 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:00.625 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:00.625 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:00.625 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:41:00.626 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:41:00.626 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:00.626 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:41:01.130 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:02.132 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:02.132 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:02.132 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:41:02.133 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:41:02.133 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:02.133 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:41:02.570 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:03.572 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:03.572 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:03.572 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:41:03.573 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:41:03.573 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:03.573 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:41:04.028 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:05.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:05.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:05.030 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:41:05.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:41:05.031 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:05.031 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:41:05.526 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:06.527 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:06.528 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:06.528 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:41:06.529 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:41:06.529 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:06.529 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:41:07.182 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:08.184 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:08.184 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:08.185 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:41:08.185 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:41:08.185 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:08.185 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:41:08.862 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:09.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:09.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:09.864 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:41:09.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:41:09.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:09.865 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:41:10.673 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:11.674 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:11.675 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:11.675 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:41:11.675 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:41:11.675 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:11.675 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:41:12.714 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:13.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:13.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:13.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:41:13.716 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:41:13.717 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:13.717 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:41:14.659 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:15.661 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:15.661 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:15.661 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:41:15.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:41:15.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:15.662 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:41:16.703 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:17.705 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:17.705 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:17.705 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:41:17.706 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:41:17.706 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:17.706 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:41:18.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:41:19.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:41:19.585 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:41:19.585 INFO:tasks.workunit.client.0.smithi139.stdout:18 2024-09-15T20:41:19.585 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:41:19.586 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:41:19.586 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:41:20.019 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/488572342,v1:127.0.0.1:6819/488572342] [v2:127.0.0.1:6820/488572342,v1:127.0.0.1:6821/488572342] exists,up e0a289bd-3c1f-48e6-b612-9a2c75ec33bb 2024-09-15T20:41:20.019 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:41:20.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:41:20.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:41:20.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2024-09-15T20:41:20.666 INFO:tasks.workunit.client.0.smithi139.stderr:noout is set 2024-09-15T20:41:20.689 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2024-09-15T20:41:21.737 INFO:tasks.workunit.client.0.smithi139.stderr:noin is set 2024-09-15T20:41:21.756 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2024-09-15T20:41:22.557 INFO:tasks.workunit.client.0.smithi139.stderr:nodown is set 2024-09-15T20:41:22.577 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2024-09-15T20:41:22.577 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=test 2024-09-15T20:41:22.578 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-15T20:41:22.578 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-15T20:41:22.578 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-15T20:41:22.578 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-15T20:41:23.308 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool test 1 1 erasure myprofile 2024-09-15T20:41:23.308 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-09-15T20:41:24.186 INFO:tasks.workunit.client.0.smithi139.stderr:pool 'test' already exists 2024-09-15T20:41:24.203 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:41:25.205 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-15T20:41:25.205 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2024-09-15T20:41:25.988 INFO:tasks.workunit.client.0.smithi139.stderr:set pool 1 allow_ec_overwrites to true 2024-09-15T20:41:26.010 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-15T20:41:26.010 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:41:26.010 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:41:26.010 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:41:26.011 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:41:26.011 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:41:26.012 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:41:26.012 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:41:26.012 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:41:26.012 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:41:26.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:41:26.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:41:26.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:41:26.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:41:26.129 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:41:26.130 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:41:26.731 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:41:26.732 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:41:26.732 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:41:26.732 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:41:26.732 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:26.732 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:41:26.904 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-15T20:41:26.904 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-15T20:41:26.904 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-15T20:41:26.904 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:26.904 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:41:27.068 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672976 2024-09-15T20:41:27.068 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672976 2024-09-15T20:41:27.068 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672976' 2024-09-15T20:41:27.068 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:27.068 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:41:27.217 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509447 2024-09-15T20:41:27.218 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509447 2024-09-15T20:41:27.218 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672976 2-64424509447' 2024-09-15T20:41:27.218 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:27.218 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-15T20:41:27.218 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:27.219 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:41:27.219 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:27.220 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-15T20:41:27.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-15T20:41:27.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-15T20:41:27.221 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 21474836504 2024-09-15T20:41:27.221 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:41:27.659 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836504 2024-09-15T20:41:27.659 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:41:28.660 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:41:28.661 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:41:29.148 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836504 2024-09-15T20:41:29.148 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:29.148 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672976 2024-09-15T20:41:29.148 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:29.150 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:41:29.151 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672976 2024-09-15T20:41:29.151 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:29.152 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672976 2024-09-15T20:41:29.152 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.1 seq 42949672976 2024-09-15T20:41:29.152 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-15T20:41:29.153 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:41:29.618 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-15T20:41:29.618 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:29.619 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509447 2024-09-15T20:41:29.619 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:29.620 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:41:29.621 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509447 2024-09-15T20:41:29.621 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:29.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509447 2024-09-15T20:41:29.622 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 64424509447 2024-09-15T20:41:29.623 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-15T20:41:29.623 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:41:30.046 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509447 2024-09-15T20:41:30.047 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:41:30.048 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:41:30.048 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:41:30.597 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:41:30.598 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:41:30.598 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:41:30.598 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:41:30.598 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:41:30.598 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:41:30.598 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:41:30.598 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:41:31.025 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:41:31.026 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:41:31.026 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:41:31.026 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:41:31.671 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:41:31.671 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:41:31.672 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:41:31.672 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-15T20:41:31.672 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2024-09-15T20:41:31.672 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:41:31.672 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:41:32.104 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:41:32.104 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:41:32.104 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:41:32.104 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:41:32.105 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:32.105 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:41:32.274 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836508 2024-09-15T20:41:32.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836508 2024-09-15T20:41:32.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508' 2024-09-15T20:41:32.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:32.275 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:41:32.425 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672979 2024-09-15T20:41:32.425 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672979 2024-09-15T20:41:32.425 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672979' 2024-09-15T20:41:32.425 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:32.425 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:41:32.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509450 2024-09-15T20:41:32.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509450 2024-09-15T20:41:32.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672979 2-64424509450' 2024-09-15T20:41:32.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:32.584 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836508 2024-09-15T20:41:32.584 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:32.585 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:41:32.586 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836508 2024-09-15T20:41:32.586 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:32.587 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836508 2024-09-15T20:41:32.587 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 21474836508 2024-09-15T20:41:32.587 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-15T20:41:32.587 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:41:33.012 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836508 2024-09-15T20:41:33.013 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:33.013 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672979 2024-09-15T20:41:33.013 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:33.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:41:33.015 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672979 2024-09-15T20:41:33.015 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:33.016 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672979 2024-09-15T20:41:33.016 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-15T20:41:33.017 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.1 seq 42949672979 2024-09-15T20:41:33.017 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:41:33.454 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-15T20:41:33.454 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:33.455 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509450 2024-09-15T20:41:33.455 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:33.456 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:41:33.457 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509450 2024-09-15T20:41:33.457 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:33.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509450 2024-09-15T20:41:33.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-09-15T20:41:33.459 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 64424509450 2024-09-15T20:41:33.459 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:41:33.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509450 -lt 64424509450 2024-09-15T20:41:33.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2024-09-15T20:41:33.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:41:33.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:41:33.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:41:33.890 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:41:33.890 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:41:33.890 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:41:33.891 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:41:33.891 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:41:33.891 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:41:33.992 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:41:33.992 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:41:33.992 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:41:33.992 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:41:33.992 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:41:33.992 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:41:34.459 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:41:34.459 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:41:34.459 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:41:34.459 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:41:34.459 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:34.459 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:41:34.631 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836510 2024-09-15T20:41:34.631 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836510 2024-09-15T20:41:34.631 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510' 2024-09-15T20:41:34.631 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:34.632 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:41:34.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672981 2024-09-15T20:41:34.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672981 2024-09-15T20:41:34.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672981' 2024-09-15T20:41:34.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:41:34.784 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:41:34.943 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509453 2024-09-15T20:41:34.943 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509453 2024-09-15T20:41:34.943 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672981 2-64424509453' 2024-09-15T20:41:34.943 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:34.944 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836510 2024-09-15T20:41:34.944 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:34.944 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:41:34.945 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836510 2024-09-15T20:41:34.945 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:34.946 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836510 2024-09-15T20:41:34.946 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-15T20:41:34.946 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 21474836510 2024-09-15T20:41:34.947 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:41:35.404 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836510 -lt 21474836510 2024-09-15T20:41:35.404 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:35.405 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672981 2024-09-15T20:41:35.405 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:35.407 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:41:35.408 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672981 2024-09-15T20:41:35.408 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:35.409 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672981 2024-09-15T20:41:35.409 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-15T20:41:35.409 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.1 seq 42949672981 2024-09-15T20:41:35.410 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:41:35.846 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672981 -lt 42949672981 2024-09-15T20:41:35.847 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:41:35.847 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509453 2024-09-15T20:41:35.847 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:41:35.849 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:41:35.849 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509453 2024-09-15T20:41:35.849 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:41:35.851 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509453 2024-09-15T20:41:35.851 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-15T20:41:35.851 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 64424509453 2024-09-15T20:41:35.851 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:41:36.299 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509451 -lt 64424509453 2024-09-15T20:41:36.299 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:41:37.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:41:37.302 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:41:37.744 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-15T20:41:37.745 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:41:37.745 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:41:37.745 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:41:38.335 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:41:38.335 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:41:38.336 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:41:38.336 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:41:38.336 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:41:38.336 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:41:38.337 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:41:38.337 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:41:38.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:41:38.760 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:41:38.760 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:41:38.760 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:41:39.356 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:41:39.356 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:41:39.356 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:41:39.357 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2024-09-15T20:41:39.357 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2024-09-15T20:41:39.766 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:41:39.783 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2024-09-15T20:41:39.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2024-09-15T20:41:39.784 INFO:tasks.workunit.client.0.smithi139.stdout:primary and soon to be divergent is 1 2024-09-15T20:41:39.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T20:41:40.184 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:41:40.184 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T20:41:25.191382+0000 0'0 29:18 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:41:23.916663+0000 0'0 2024-09-15T20:41:23.916663+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:49:57.933107+0000 0 0 2024-09-15T20:41:40.184 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:41:40.184 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:41:40.184 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:41:40.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2024-09-15T20:41:40.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-15T20:41:40.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2024-09-15T20:41:40.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2024-09-15T20:41:40.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-15T20:41:40.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2024-09-15T20:41:40.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2024-09-15T20:41:40.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-15T20:41:40.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2024-09-15T20:41:40.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2024-09-15T20:41:40.201 INFO:tasks.workunit.client.0.smithi139.stdout:writing initial objects 2024-09-15T20:41:40.202 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2024-09-15T20:41:40.202 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2024-09-15T20:41:40.203 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:40.203 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2024-09-15T20:41:40.369 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:40.369 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2024-09-15T20:41:40.629 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:40.629 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2024-09-15T20:41:40.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:40.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2024-09-15T20:41:41.108 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:41.108 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2024-09-15T20:41:41.355 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:41.356 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2024-09-15T20:41:41.611 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:41.612 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2024-09-15T20:41:41.845 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:41.845 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2024-09-15T20:41:42.088 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:42.088 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2024-09-15T20:41:42.314 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:42.314 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2024-09-15T20:41:42.566 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:42.566 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2024-09-15T20:41:42.842 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:42.842 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2024-09-15T20:41:43.087 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:43.087 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2024-09-15T20:41:43.342 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:43.342 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2024-09-15T20:41:43.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:43.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2024-09-15T20:41:43.839 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:43.839 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2024-09-15T20:41:44.099 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:44.100 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2024-09-15T20:41:44.318 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:44.318 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2024-09-15T20:41:44.552 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:44.552 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2024-09-15T20:41:44.778 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:44.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2024-09-15T20:41:44.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:44.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2024-09-15T20:41:45.240 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:45.240 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2024-09-15T20:41:45.517 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:45.517 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2024-09-15T20:41:45.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:45.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2024-09-15T20:41:46.036 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:46.036 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2024-09-15T20:41:46.262 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:46.262 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2024-09-15T20:41:46.472 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:46.472 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2024-09-15T20:41:46.681 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:46.682 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2024-09-15T20:41:46.940 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:46.941 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2024-09-15T20:41:47.184 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:47.184 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2024-09-15T20:41:47.462 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:47.462 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2024-09-15T20:41:47.696 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:47.696 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2024-09-15T20:41:47.914 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:47.914 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2024-09-15T20:41:48.106 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:48.106 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2024-09-15T20:41:48.323 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:48.323 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2024-09-15T20:41:48.575 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:48.575 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2024-09-15T20:41:48.835 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:48.835 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2024-09-15T20:41:49.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:49.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2024-09-15T20:41:49.396 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:49.396 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2024-09-15T20:41:49.648 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:49.648 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2024-09-15T20:41:49.865 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:49.865 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2024-09-15T20:41:50.110 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:50.110 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2024-09-15T20:41:50.351 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:50.352 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2024-09-15T20:41:50.628 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:50.628 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2024-09-15T20:41:50.871 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:50.872 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2024-09-15T20:41:51.098 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:51.098 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2024-09-15T20:41:51.307 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:51.308 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2024-09-15T20:41:51.534 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:51.534 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2024-09-15T20:41:51.751 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:51.752 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2024-09-15T20:41:52.004 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:52.004 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2024-09-15T20:41:52.313 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:52.313 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2024-09-15T20:41:52.539 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:52.540 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2024-09-15T20:41:52.790 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:52.791 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2024-09-15T20:41:53.076 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:53.076 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2024-09-15T20:41:53.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:53.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2024-09-15T20:41:53.546 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:53.547 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2024-09-15T20:41:53.755 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:53.755 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2024-09-15T20:41:53.956 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:53.956 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2024-09-15T20:41:54.199 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:54.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2024-09-15T20:41:54.483 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:54.483 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2024-09-15T20:41:54.744 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:54.744 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2024-09-15T20:41:55.011 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:55.012 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2024-09-15T20:41:55.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:55.297 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2024-09-15T20:41:55.549 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:55.549 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2024-09-15T20:41:55.793 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:55.793 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2024-09-15T20:41:56.051 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:56.051 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2024-09-15T20:41:56.318 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:56.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2024-09-15T20:41:56.554 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:56.554 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2024-09-15T20:41:56.796 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:56.796 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2024-09-15T20:41:56.997 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:56.997 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2024-09-15T20:41:57.232 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:57.232 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2024-09-15T20:41:57.468 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:57.468 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2024-09-15T20:41:57.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:57.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2024-09-15T20:41:57.953 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:57.953 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2024-09-15T20:41:58.187 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:58.187 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2024-09-15T20:41:58.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:58.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2024-09-15T20:41:58.616 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:58.616 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2024-09-15T20:41:58.850 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:58.850 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2024-09-15T20:41:59.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:59.128 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2024-09-15T20:41:59.414 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:59.414 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2024-09-15T20:41:59.655 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:59.656 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2024-09-15T20:41:59.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:41:59.890 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2024-09-15T20:42:00.158 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:42:00.158 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2024-09-15T20:42:00.375 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:42:00.375 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2024-09-15T20:42:00.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:42:00.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2024-09-15T20:42:00.811 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:42:00.811 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2024-09-15T20:42:01.046 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:42:01.046 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2024-09-15T20:42:01.297 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:42:01.297 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2024-09-15T20:42:01.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:42:01.524 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2024-09-15T20:42:01.775 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:42:01.775 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2024-09-15T20:42:02.068 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:42:02.069 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2024-09-15T20:42:02.479 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:42:02.479 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2024-09-15T20:42:02.696 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:42:02.696 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2024-09-15T20:42:02.931 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:42:02.931 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2024-09-15T20:42:03.207 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:42:03.207 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2024-09-15T20:42:03.492 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:42:03.493 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2024-09-15T20:42:03.752 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:42:03.752 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2024-09-15T20:42:03.979 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:42:03.979 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2024-09-15T20:42:04.197 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:42:04.197 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-09-15T20:42:04.390 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T20:42:04.390 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-09-15T20:42:04.599 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T20:42:04.599 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2024-09-15T20:42:04.599 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:42:04.599 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:42:04.599 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:42:04.600 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T20:42:04.601 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:42:04.601 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:42:04.601 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:42:04.601 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:42:04.601 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:42:04.678 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T20:42:04.678 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:42:04.678 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:42:04.678 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:42:04.678 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:42:04.679 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:42:05.168 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:42:05.169 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:42:05.169 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:42:05.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:42:05.169 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:42:05.169 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:42:05.352 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836518 2024-09-15T20:42:05.352 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836518 2024-09-15T20:42:05.352 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836518' 2024-09-15T20:42:05.352 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:42:05.353 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:42:05.517 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672989 2024-09-15T20:42:05.517 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672989 2024-09-15T20:42:05.517 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836518 1-42949672989' 2024-09-15T20:42:05.517 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:42:05.517 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:42:05.681 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509461 2024-09-15T20:42:05.681 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509461 2024-09-15T20:42:05.681 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836518 1-42949672989 2-64424509461' 2024-09-15T20:42:05.681 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:42:05.682 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836518 2024-09-15T20:42:05.682 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:42:05.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:42:05.684 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836518 2024-09-15T20:42:05.684 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:42:05.685 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836518 2024-09-15T20:42:05.685 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 21474836518 2024-09-15T20:42:05.685 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-09-15T20:42:05.686 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:42:06.115 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836516 -lt 21474836518 2024-09-15T20:42:06.116 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:42:07.117 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:42:07.117 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:42:07.556 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836518 -lt 21474836518 2024-09-15T20:42:07.557 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:42:07.557 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672989 2024-09-15T20:42:07.557 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:42:07.558 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:42:07.560 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:42:07.560 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672989 2024-09-15T20:42:07.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672989 2024-09-15T20:42:07.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-09-15T20:42:07.561 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.1 seq 42949672989 2024-09-15T20:42:07.561 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:42:08.053 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672990 -lt 42949672989 2024-09-15T20:42:08.053 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:42:08.054 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:42:08.054 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509461 2024-09-15T20:42:08.055 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:42:08.056 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509461 2024-09-15T20:42:08.056 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:42:08.057 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509461 2024-09-15T20:42:08.058 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 64424509461 2024-09-15T20:42:08.058 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-15T20:42:08.058 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:42:08.499 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509461 -lt 64424509461 2024-09-15T20:42:08.499 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:42:08.500 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:42:08.500 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:42:09.087 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:42:09.087 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:42:09.087 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:42:09.087 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:42:09.088 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:42:09.088 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:42:09.088 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:42:09.088 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:42:09.528 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:42:09.528 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:42:09.528 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:42:09.529 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:42:10.075 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:42:10.075 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:42:10.075 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:42:10.076 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2024-09-15T20:42:10.076 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-15T20:42:10.076 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-15T20:42:10.077 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-15T20:42:10.077 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-15T20:42:10.537 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2024-09-15T20:42:10.537 INFO:tasks.workunit.client.0.smithi139.stdout:blackholing osds 0 2 2024-09-15T20:42:10.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2024-09-15T20:42:10.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T20:42:10.955 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:42:10.955 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-09-15T20:41:25.191382+0000 29'100 29:118 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:41:23.916663+0000 0'0 2024-09-15T20:41:23.916663+0000 0 0 periodic scrub scheduled @ 2024-09-16T20:49:57.933107+0000 0 0 2024-09-15T20:42:10.956 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:42:10.956 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:42:10.956 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:42:10.971 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-09-15T20:42:10.972 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2024-09-15T20:42:10.972 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T20:42:10.972 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T20:42:10.973 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:42:10.973 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:42:10.973 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:42:10.973 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.0.asok 2024-09-15T20:42:10.973 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-09-15T20:42:10.973 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-15T20:42:11.071 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:42:11.071 INFO:tasks.workunit.client.0.smithi139.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-15T20:42:11.071 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:42:11.083 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-09-15T20:42:11.083 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2024-09-15T20:42:11.083 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-15T20:42:11.083 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-15T20:42:11.084 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:42:11.084 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:42:11.084 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:42:11.085 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.2.asok 2024-09-15T20:42:11.085 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-09-15T20:42:11.085 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-15T20:42:11.182 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:42:11.182 INFO:tasks.workunit.client.0.smithi139.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-15T20:42:11.182 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:42:11.194 INFO:tasks.workunit.client.0.smithi139.stdout:writing divergent object 2024-09-15T20:42:11.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2024-09-15T20:42:11.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2024-09-15T20:42:11.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2024-09-15T20:42:12.197 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2024-09-15T20:42:12.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-09-15T20:42:12.431 INFO:tasks.workunit.client.0.smithi139.stdout:created pool test snap snap1 2024-09-15T20:42:12.436 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2024-09-15T20:42:12.436 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2024-09-15T20:42:12.438 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-09-15T20:42:22.439 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2024-09-15T20:42:22.443 INFO:tasks.workunit.client.0.smithi139.stderr:rados: no process found 2024-09-15T20:42:22.443 INFO:tasks.workunit.client.0.smithi139.stdout:killing all the osds 2024-09-15T20:42:22.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2024-09-15T20:42:22.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T20:42:22.873 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:42:22.873 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 active+clean 2024-09-15T20:41:25.191382+0000 30'104 30:124 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:41:23.916663+0000 0'0 2024-09-15T20:41:23.916663+0000 0 0 periodic scrub scheduled @ 2024-09-17T01:45:10.447261+0000 0 0 2024-09-15T20:42:22.873 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:42:22.873 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:42:22.873 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:42:22.890 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2024-09-15T20:42:22.891 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:42:22.891 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:42:22.892 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:42:22.892 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:42:22.892 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:42:23.002 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:42:23.002 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-15T20:42:23.002 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2024-09-15T20:42:23.620 INFO:tasks.workunit.client.0.smithi139.stderr:osd.0 is already down. 2024-09-15T20:42:23.636 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-15T20:42:23.636 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2024-09-15T20:42:24.221 INFO:tasks.workunit.client.0.smithi139.stderr:osd.1 is already down. 2024-09-15T20:42:24.239 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-15T20:42:24.240 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2024-09-15T20:42:24.839 INFO:tasks.workunit.client.0.smithi139.stderr:osd.2 is already down. 2024-09-15T20:42:24.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-09-15T20:42:24.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2024-09-15T20:42:25.513 INFO:tasks.workunit.client.0.smithi139.stderr:osd.0 is already out. 2024-09-15T20:42:25.529 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-09-15T20:42:25.530 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2024-09-15T20:42:26.110 INFO:tasks.workunit.client.0.smithi139.stderr:osd.2 is already out. 2024-09-15T20:42:26.126 INFO:tasks.workunit.client.0.smithi139.stdout:bringing up non_divergent 0 2 2024-09-15T20:42:26.127 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2024-09-15T20:42:26.127 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T20:42:26.561 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:42:26.562 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 stale+active+clean 2024-09-15T20:41:25.191382+0000 30'104 30:124 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:41:23.916663+0000 0'0 2024-09-15T20:41:23.916663+0000 0 0 periodic scrub scheduled @ 2024-09-17T01:45:10.447261+0000 0 0 2024-09-15T20:42:26.562 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:42:26.562 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:42:26.562 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:42:26.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-09-15T20:42:26.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-09-15T20:42:26.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-15T20:42:26.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:42:26.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:42:26.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:42:26.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-15T20:42:26.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:42:26.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:42:26.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:42:26.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:42:26.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-15T20:42:26.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-15T20:42:26.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:42:26.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:42:26.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:42:26.581 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:42:26.581 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:42:26.581 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:42:26.582 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:42:26.582 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:42:26.582 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:42:26.582 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:42:26.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:42:26.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:42:26.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:42:26.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:42:26.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:42:26.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:42:26.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:42:26.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:42:26.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:42:26.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:42:26.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-15T20:42:26.583 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T20:42:26.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:42:26.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:42:26.584 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-15T20:42:26.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:42:26.585 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:42:26.585 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:42:26.585 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:42:26.623 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:42:26.623+0000 7f9ea70a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:42:26.625 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:42:26.627+0000 7f9ea70a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:42:26.626 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:42:26.627+0000 7f9ea70a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:42:27.070 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:42:27.070 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:42:27.070 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:42:27.070 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:42:27.070 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:42:27.070 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:27.070 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:42:27.071 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:42:27.071 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:27.071 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:42:27.538 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:27.880 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:42:27.879+0000 7f9ea70a27c0 -1 Falling back to public interface 2024-09-15T20:42:28.539 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:28.540 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:28.540 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:42:28.540 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:42:28.540 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:28.541 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:42:29.013 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:30.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:30.014 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:30.014 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:42:30.015 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:42:30.015 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:30.015 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:42:30.471 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:31.180 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:42:31.179+0000 7f9ea70a27c0 -1 osd.0 29 log_to_monitors true 2024-09-15T20:42:31.473 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:31.473 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:31.473 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:42:31.474 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:42:31.474 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:31.474 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:42:32.014 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/1240929079,v1:127.0.0.1:6803/1240929079] [v2:127.0.0.1:6804/1240929079,v1:127.0.0.1:6805/1240929079] exists,up 072ef209-0424-4dcb-ba88-59a19eaa46bd 2024-09-15T20:42:32.016 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:42:32.016 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:42:32.016 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:42:32.017 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-09-15T20:42:32.017 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-09-15T20:42:32.017 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-15T20:42:32.017 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:42:32.017 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T20:42:32.017 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:42:32.017 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-15T20:42:32.017 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:42:32.017 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:42:32.017 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:42:32.017 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:42:32.017 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-15T20:42:32.017 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-15T20:42:32.018 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:42:32.018 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:42:32.018 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:42:32.018 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:42:32.018 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:42:32.018 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:42:32.018 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:42:32.018 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:42:32.018 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:42:32.018 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:42:32.019 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:42:32.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:42:32.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:42:32.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:42:32.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:42:32.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:42:32.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:42:32.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:42:32.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:42:32.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:42:32.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-15T20:42:32.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T20:42:32.021 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.2 2024-09-15T20:42:32.021 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:42:32.021 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-15T20:42:32.022 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T20:42:32.022 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:42:32.022 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:42:32.022 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:42:32.055 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:42:32.054+0000 7ff0db7b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:42:32.057 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:42:32.058+0000 7ff0db7b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:42:32.058 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:42:32.058+0000 7ff0db7b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:42:32.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T20:42:32.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:42:32.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:42:32.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:42:32.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:42:32.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:32.445 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:42:32.445 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:42:32.445 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:32.445 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:42:32.950 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:33.352 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:42:33.350+0000 7ff0db7b47c0 -1 Falling back to public interface 2024-09-15T20:42:33.951 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:33.952 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:33.952 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:42:33.952 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:42:33.953 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:33.953 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:42:34.445 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:35.446 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:35.447 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:35.447 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:42:35.447 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:42:35.447 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:35.448 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:42:35.905 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:42:36.698 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:42:36.698+0000 7ff0db7b47c0 -1 osd.2 29 log_to_monitors true 2024-09-15T20:42:36.906 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:42:36.906 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:42:36.906 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:42:36.907 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:42:36.907 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:42:36.907 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:42:37.429 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up out weight 0 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/85082059,v1:127.0.0.1:6811/85082059] [v2:127.0.0.1:6812/85082059,v1:127.0.0.1:6813/85082059] exists,up e0a289bd-3c1f-48e6-b612-9a2c75ec33bb 2024-09-15T20:42:37.430 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:42:37.430 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:42:37.430 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:42:37.430 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-09-15T20:42:37.430 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2024-09-15T20:42:38.020 INFO:tasks.workunit.client.0.smithi139.stderr:osd.0 is already in. 2024-09-15T20:42:38.036 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-09-15T20:42:38.037 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2024-09-15T20:42:38.885 INFO:tasks.workunit.client.0.smithi139.stderr:osd.2 is already in. 2024-09-15T20:42:38.906 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2024-09-15T20:42:43.909 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2024-09-15T20:42:43.910 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2024-09-15T20:42:43.910 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2024-09-15T20:42:43.910 INFO:tasks.workunit.client.0.smithi139.stdout:writing non-divergent object existing_4 2024-09-15T20:42:43.910 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T20:42:44.320 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:42:44.320 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-15T20:42:39.119858+0000 29'100 43:129 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-15T20:41:23.916663+0000 0'0 2024-09-15T20:41:23.916663+0000 0 0 periodic scrub scheduled @ 2024-09-17T05:49:30.197344+0000 0 0 2024-09-15T20:42:44.321 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:42:44.321 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:42:44.321 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:42:44.336 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2024-09-15T20:42:44.449 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T20:42:44.449 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2024-09-15T20:42:44.449 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:42:44.449 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:42:44.449 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:42:44.449 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T20:42:44.450 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:42:44.450 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:42:44.451 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:42:44.451 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:42:44.451 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:42:44.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T20:42:44.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:42:44.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:42:44.520 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:42:44.520 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:42:44.520 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:42:44.971 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:42:44.971 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:42:44.971 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:42:44.971 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:42:44.971 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:42:44.972 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:42:45.145 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=158913789957 2024-09-15T20:42:45.145 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 158913789957 2024-09-15T20:42:45.146 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-158913789957' 2024-09-15T20:42:45.146 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:42:45.146 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:42:45.259 INFO:tasks.workunit.client.0.smithi139.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-15T20:42:45.261 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-15T20:42:45.262 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-15T20:42:45.262 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-15T20:42:45.262 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:42:45.262 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:42:45.413 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=167503724548 2024-09-15T20:42:45.413 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 167503724548 2024-09-15T20:42:45.413 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-158913789957 2-167503724548' 2024-09-15T20:42:45.413 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:42:45.414 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-158913789957 2024-09-15T20:42:45.414 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:42:45.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:42:45.415 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-158913789957 2024-09-15T20:42:45.415 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:42:45.416 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=158913789957 2024-09-15T20:42:45.416 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 158913789957 2024-09-15T20:42:45.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 158913789957' 2024-09-15T20:42:45.417 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:42:45.843 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 158913789955 -lt 158913789957 2024-09-15T20:42:45.844 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:42:46.845 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:42:46.845 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:42:47.272 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 158913789958 -lt 158913789957 2024-09-15T20:42:47.272 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:42:47.273 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:42:47.273 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-167503724548 2024-09-15T20:42:47.274 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:42:47.275 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-167503724548 2024-09-15T20:42:47.275 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:42:47.276 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=167503724548 2024-09-15T20:42:47.277 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 167503724548' 2024-09-15T20:42:47.277 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 167503724548 2024-09-15T20:42:47.277 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:42:47.723 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724548 -lt 167503724548 2024-09-15T20:42:47.724 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:42:47.725 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:42:47.725 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:42:48.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:42:48.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:42:48.303 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:42:48.303 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:42:48.303 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:42:48.303 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:42:48.303 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:42:48.303 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:42:48.753 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:42:48.754 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:42:48.754 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:42:48.754 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:42:49.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:42:49.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T20:42:49.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:42:49.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T20:42:49.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:42:49.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:42:49.402 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:42:49.402 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:42:49.403 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:42:49.403 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:42:49.403 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:42:49.403 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:42:49.403 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:42:49.404 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:42:49.819 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:42:49.820 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:42:49.820 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:42:49.821 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:42:50.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:42:50.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:42:50.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:42:50.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:42:50.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:42:50.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:42:50.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:42:50.416 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:42:50.416 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:42:50.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:42:50.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:42:50.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-15T20:42:50.991 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:42:50.991 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:42:51.191 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:42:51.191 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:42:51.192 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:42:51.192 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:42:51.192 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:42:51.192 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:42:51.193 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:42:51.193 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:42:51.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:42:51.621 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:42:51.622 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:42:51.622 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:42:52.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:42:52.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:42:52.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:42:52.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:42:52.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:42:52.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:42:52.173 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:42:52.174 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:42:52.174 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:42:52.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:42:52.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:42:52.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-15T20:42:52.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:42:52.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T20:42:53.188 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:42:53.188 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:42:53.189 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:42:53.189 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:42:53.189 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:42:53.189 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:42:53.189 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:42:53.189 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:42:53.624 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:42:53.624 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:42:53.624 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:42:53.624 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:42:54.180 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:42:54.180 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:42:54.180 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:42:54.180 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:42:54.180 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:42:54.181 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:42:54.181 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:42:54.181 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:42:54.181 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:42:54.753 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:42:54.753 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:42:54.753 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-15T20:42:54.753 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:42:54.753 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-15T20:42:55.554 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:42:55.554 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:42:55.555 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:42:55.555 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:42:55.555 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:42:55.555 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:42:55.556 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:42:55.556 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:42:55.981 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:42:55.982 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:42:55.983 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:42:55.983 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:42:56.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:42:56.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:42:56.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:42:56.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:42:56.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:42:56.562 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:42:56.562 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:42:56.562 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:42:56.562 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:42:57.179 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:42:57.179 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:42:57.179 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-15T20:42:57.179 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:42:57.179 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-15T20:42:58.780 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:42:58.780 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:42:58.781 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:42:58.781 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:42:58.781 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:42:58.781 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:42:58.781 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:42:58.782 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:42:59.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:42:59.242 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:42:59.242 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:42:59.242 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:42:59.834 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:42:59.834 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:42:59.834 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:42:59.834 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:42:59.834 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:42:59.834 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:42:59.834 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:42:59.835 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:42:59.835 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:43:00.491 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:43:00.491 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:43:00.491 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-15T20:43:00.491 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:43:00.492 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-15T20:43:03.693 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:43:03.693 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:43:03.694 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:43:03.694 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:43:03.694 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:43:03.694 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:43:03.694 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:43:03.695 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:43:04.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:43:04.122 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:43:04.122 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:43:04.123 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:43:04.687 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:43:04.688 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:43:04.688 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:43:04.688 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:43:04.688 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:43:04.688 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:43:04.688 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:43:04.688 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:43:04.689 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:43:05.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:43:05.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:43:05.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-15T20:43:05.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:43:05.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-15T20:43:11.703 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:43:11.703 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:43:11.704 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:43:11.704 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:43:11.704 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:43:11.704 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:43:11.704 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:43:11.705 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:43:12.134 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:43:12.134 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:43:12.135 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:43:12.135 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:43:12.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:43:12.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:43:12.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:43:12.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:43:12.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:43:12.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:43:12.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:43:12.796 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:43:12.796 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:43:13.357 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:43:13.358 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:43:13.358 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-15T20:43:13.358 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:43:13.358 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-15T20:43:20.659 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:43:20.659 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:43:20.660 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:43:20.660 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:43:20.660 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:43:20.660 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:43:20.661 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:43:20.661 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:43:21.112 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:43:21.113 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:43:21.113 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:43:21.113 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:43:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:43:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:43:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:43:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:43:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:43:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:43:21.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:43:21.684 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:43:21.684 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:43:22.261 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:43:22.261 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:43:22.261 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-15T20:43:22.261 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-15T20:43:22.787 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:43:22.787 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_fingerprint": "5ac362c1-a7e8-4310-8ba0-1d9708f202c3", 2024-09-15T20:43:22.787 INFO:tasks.workunit.client.0.smithi139.stdout: "version": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:43:22.787 INFO:tasks.workunit.client.0.smithi139.stdout: "commit": "fc3e0bff7468fd55478adbae7f75508b749124b6", 2024-09-15T20:43:22.787 INFO:tasks.workunit.client.0.smithi139.stdout: "timestamp": "2024-09-15T20:43:22.790789+0000", 2024-09-15T20:43:22.787 INFO:tasks.workunit.client.0.smithi139.stdout: "tag": "", 2024-09-15T20:43:22.787 INFO:tasks.workunit.client.0.smithi139.stdout: "health": { 2024-09-15T20:43:22.788 INFO:tasks.workunit.client.0.smithi139.stdout: "status": "HEALTH_WARN", 2024-09-15T20:43:22.788 INFO:tasks.workunit.client.0.smithi139.stdout: "checks": { 2024-09-15T20:43:22.788 INFO:tasks.workunit.client.0.smithi139.stdout: "OSDMAP_FLAGS": { 2024-09-15T20:43:22.788 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:43:22.788 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:43:22.788 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-15T20:43:22.788 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 17 2024-09-15T20:43:22.788 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.788 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [], 2024-09-15T20:43:22.788 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:43:22.788 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.788 INFO:tasks.workunit.client.0.smithi139.stdout: "OSD_DOWN": { 2024-09-15T20:43:22.789 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:43:22.789 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:43:22.789 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "1 osds down", 2024-09-15T20:43:22.789 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 1 2024-09-15T20:43:22.789 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.789 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:43:22.789 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.789 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "osd.1 (root=default,host=smithi139) is down" 2024-09-15T20:43:22.789 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.789 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.789 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:43:22.789 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.790 INFO:tasks.workunit.client.0.smithi139.stdout: "PG_DEGRADED": { 2024-09-15T20:43:22.790 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:43:22.790 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:43:22.790 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-09-15T20:43:22.790 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 1 2024-09-15T20:43:22.790 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.790 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:43:22.790 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.790 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-09-15T20:43:22.790 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.790 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.790 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:43:22.790 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.790 INFO:tasks.workunit.client.0.smithi139.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-15T20:43:22.791 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:43:22.791 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:43:22.791 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-15T20:43:22.791 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 1 2024-09-15T20:43:22.791 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.791 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:43:22.791 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.791 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "application not enabled on pool 'test'" 2024-09-15T20:43:22.791 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.791 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.791 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-15T20:43:22.791 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.791 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.791 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:43:22.792 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.792 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.792 INFO:tasks.workunit.client.0.smithi139.stdout: "mutes": [] 2024-09-15T20:43:22.792 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.792 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap_first_committed": 1, 2024-09-15T20:43:22.792 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap_last_committed": 1, 2024-09-15T20:43:22.792 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap": { 2024-09-15T20:43:22.792 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 1, 2024-09-15T20:43:22.792 INFO:tasks.workunit.client.0.smithi139.stdout: "fsid": "0e220df1-ac9c-433e-8885-cf9f7aab293e", 2024-09-15T20:43:22.792 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:39:24.243736Z", 2024-09-15T20:43:22.792 INFO:tasks.workunit.client.0.smithi139.stdout: "created": "2024-09-15T20:39:24.243736Z", 2024-09-15T20:43:22.793 INFO:tasks.workunit.client.0.smithi139.stdout: "min_mon_release": 19, 2024-09-15T20:43:22.793 INFO:tasks.workunit.client.0.smithi139.stdout: "min_mon_release_name": "squid", 2024-09-15T20:43:22.793 INFO:tasks.workunit.client.0.smithi139.stdout: "election_strategy": 1, 2024-09-15T20:43:22.793 INFO:tasks.workunit.client.0.smithi139.stdout: "disallowed_leaders: ": "", 2024-09-15T20:43:22.793 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode": false, 2024-09-15T20:43:22.793 INFO:tasks.workunit.client.0.smithi139.stdout: "tiebreaker_mon": "", 2024-09-15T20:43:22.793 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_ranks: ": "", 2024-09-15T20:43:22.793 INFO:tasks.workunit.client.0.smithi139.stdout: "features": { 2024-09-15T20:43:22.793 INFO:tasks.workunit.client.0.smithi139.stdout: "persistent": [ 2024-09-15T20:43:22.793 INFO:tasks.workunit.client.0.smithi139.stdout: "kraken", 2024-09-15T20:43:22.793 INFO:tasks.workunit.client.0.smithi139.stdout: "luminous", 2024-09-15T20:43:22.793 INFO:tasks.workunit.client.0.smithi139.stdout: "mimic", 2024-09-15T20:43:22.793 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap-prune", 2024-09-15T20:43:22.793 INFO:tasks.workunit.client.0.smithi139.stdout: "nautilus", 2024-09-15T20:43:22.794 INFO:tasks.workunit.client.0.smithi139.stdout: "octopus", 2024-09-15T20:43:22.794 INFO:tasks.workunit.client.0.smithi139.stdout: "pacific", 2024-09-15T20:43:22.794 INFO:tasks.workunit.client.0.smithi139.stdout: "elector-pinging", 2024-09-15T20:43:22.794 INFO:tasks.workunit.client.0.smithi139.stdout: "quincy", 2024-09-15T20:43:22.794 INFO:tasks.workunit.client.0.smithi139.stdout: "reef", 2024-09-15T20:43:22.794 INFO:tasks.workunit.client.0.smithi139.stdout: "squid" 2024-09-15T20:43:22.794 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.794 INFO:tasks.workunit.client.0.smithi139.stdout: "optional": [] 2024-09-15T20:43:22.794 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.794 INFO:tasks.workunit.client.0.smithi139.stdout: "mons": [ 2024-09-15T20:43:22.794 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.794 INFO:tasks.workunit.client.0.smithi139.stdout: "rank": 0, 2024-09-15T20:43:22.795 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "a", 2024-09-15T20:43:22.795 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:43:22.795 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:43:22.795 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.795 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:43:22.795 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:7115", 2024-09-15T20:43:22.795 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 0 2024-09-15T20:43:22.795 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.795 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.795 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.795 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:7115/0", 2024-09-15T20:43:22.795 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-15T20:43:22.795 INFO:tasks.workunit.client.0.smithi139.stdout: "priority": 0, 2024-09-15T20:43:22.796 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 0, 2024-09-15T20:43:22.796 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_location": "{}" 2024-09-15T20:43:22.796 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.796 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.796 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.796 INFO:tasks.workunit.client.0.smithi139.stdout: "quorum": [ 2024-09-15T20:43:22.796 INFO:tasks.workunit.client.0.smithi139.stdout: 0 2024-09-15T20:43:22.796 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.796 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap": { 2024-09-15T20:43:22.796 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 43, 2024-09-15T20:43:22.796 INFO:tasks.workunit.client.0.smithi139.stdout: "fsid": "0e220df1-ac9c-433e-8885-cf9f7aab293e", 2024-09-15T20:43:22.796 INFO:tasks.workunit.client.0.smithi139.stdout: "created": "2024-09-15T20:39:25.208770+0000", 2024-09-15T20:43:22.796 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:42:38.660909+0000", 2024-09-15T20:43:22.797 INFO:tasks.workunit.client.0.smithi139.stdout: "last_up_change": "2024-09-15T20:42:37.012476+0000", 2024-09-15T20:43:22.797 INFO:tasks.workunit.client.0.smithi139.stdout: "last_in_change": "2024-09-15T20:42:38.459878+0000", 2024-09-15T20:43:22.797 INFO:tasks.workunit.client.0.smithi139.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T20:43:22.797 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_num": 5800384, 2024-09-15T20:43:22.797 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_set": [ 2024-09-15T20:43:22.797 INFO:tasks.workunit.client.0.smithi139.stdout: "nodown", 2024-09-15T20:43:22.797 INFO:tasks.workunit.client.0.smithi139.stdout: "noin", 2024-09-15T20:43:22.797 INFO:tasks.workunit.client.0.smithi139.stdout: "noout", 2024-09-15T20:43:22.797 INFO:tasks.workunit.client.0.smithi139.stdout: "pglog_hardlimit", 2024-09-15T20:43:22.797 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snapdirs", 2024-09-15T20:43:22.797 INFO:tasks.workunit.client.0.smithi139.stdout: "recovery_deletes", 2024-09-15T20:43:22.798 INFO:tasks.workunit.client.0.smithi139.stdout: "sortbitwise" 2024-09-15T20:43:22.798 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.798 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_version": 10, 2024-09-15T20:43:22.798 INFO:tasks.workunit.client.0.smithi139.stdout: "full_ratio": 0.99000000953674316, 2024-09-15T20:43:22.798 INFO:tasks.workunit.client.0.smithi139.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-15T20:43:22.798 INFO:tasks.workunit.client.0.smithi139.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T20:43:22.798 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_snapshot": "", 2024-09-15T20:43:22.798 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_max": 1, 2024-09-15T20:43:22.798 INFO:tasks.workunit.client.0.smithi139.stdout: "max_osd": 3, 2024-09-15T20:43:22.798 INFO:tasks.workunit.client.0.smithi139.stdout: "require_min_compat_client": "luminous", 2024-09-15T20:43:22.798 INFO:tasks.workunit.client.0.smithi139.stdout: "min_compat_client": "jewel", 2024-09-15T20:43:22.798 INFO:tasks.workunit.client.0.smithi139.stdout: "require_osd_release": "squid", 2024-09-15T20:43:22.798 INFO:tasks.workunit.client.0.smithi139.stdout: "allow_crimson": false, 2024-09-15T20:43:22.798 INFO:tasks.workunit.client.0.smithi139.stdout: "pools": [ 2024-09-15T20:43:22.799 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.799 INFO:tasks.workunit.client.0.smithi139.stdout: "pool": 1, 2024-09-15T20:43:22.799 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_name": "test", 2024-09-15T20:43:22.799 INFO:tasks.workunit.client.0.smithi139.stdout: "create_time": "2024-09-15T20:41:23.859990+0000", 2024-09-15T20:43:22.799 INFO:tasks.workunit.client.0.smithi139.stdout: "flags": 16389, 2024-09-15T20:43:22.799 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-09-15T20:43:22.799 INFO:tasks.workunit.client.0.smithi139.stdout: "type": 3, 2024-09-15T20:43:22.799 INFO:tasks.workunit.client.0.smithi139.stdout: "size": 3, 2024-09-15T20:43:22.799 INFO:tasks.workunit.client.0.smithi139.stdout: "min_size": 2, 2024-09-15T20:43:22.799 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_rule": 1, 2024-09-15T20:43:22.799 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_count": 0, 2024-09-15T20:43:22.800 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_target": 0, 2024-09-15T20:43:22.800 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T20:43:22.800 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T20:43:22.800 INFO:tasks.workunit.client.0.smithi139.stdout: "is_stretch_pool": false, 2024-09-15T20:43:22.800 INFO:tasks.workunit.client.0.smithi139.stdout: "object_hash": 2, 2024-09-15T20:43:22.800 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_autoscale_mode": "off", 2024-09-15T20:43:22.800 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num": 1, 2024-09-15T20:43:22.800 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_placement_num": 1, 2024-09-15T20:43:22.800 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_placement_num_target": 1, 2024-09-15T20:43:22.800 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num_target": 1, 2024-09-15T20:43:22.800 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num_pending": 1, 2024-09-15T20:43:22.800 INFO:tasks.workunit.client.0.smithi139.stdout: "last_pg_merge_meta": { 2024-09-15T20:43:22.800 INFO:tasks.workunit.client.0.smithi139.stdout: "source_pgid": "0.0", 2024-09-15T20:43:22.801 INFO:tasks.workunit.client.0.smithi139.stdout: "ready_epoch": 0, 2024-09-15T20:43:22.801 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_started": 0, 2024-09-15T20:43:22.801 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_clean": 0, 2024-09-15T20:43:22.801 INFO:tasks.workunit.client.0.smithi139.stdout: "source_version": "0'0", 2024-09-15T20:43:22.801 INFO:tasks.workunit.client.0.smithi139.stdout: "target_version": "0'0" 2024-09-15T20:43:22.801 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.801 INFO:tasks.workunit.client.0.smithi139.stdout: "last_change": "30", 2024-09-15T20:43:22.801 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend": "0", 2024-09-15T20:43:22.801 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T20:43:22.801 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T20:43:22.801 INFO:tasks.workunit.client.0.smithi139.stdout: "auid": 0, 2024-09-15T20:43:22.801 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_mode": "pool", 2024-09-15T20:43:22.802 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_seq": 1, 2024-09-15T20:43:22.802 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_epoch": 30, 2024-09-15T20:43:22.802 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_snaps": [ 2024-09-15T20:43:22.802 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.802 INFO:tasks.workunit.client.0.smithi139.stdout: "snapid": 1, 2024-09-15T20:43:22.802 INFO:tasks.workunit.client.0.smithi139.stdout: "stamp": "2024-09-15T20:42:12.247870+0000", 2024-09-15T20:43:22.802 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "snap1" 2024-09-15T20:43:22.802 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.802 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.803 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_snaps": "[]", 2024-09-15T20:43:22.803 INFO:tasks.workunit.client.0.smithi139.stdout: "quota_max_bytes": 0, 2024-09-15T20:43:22.803 INFO:tasks.workunit.client.0.smithi139.stdout: "quota_max_objects": 0, 2024-09-15T20:43:22.803 INFO:tasks.workunit.client.0.smithi139.stdout: "tiers": [], 2024-09-15T20:43:22.803 INFO:tasks.workunit.client.0.smithi139.stdout: "tier_of": -1, 2024-09-15T20:43:22.803 INFO:tasks.workunit.client.0.smithi139.stdout: "read_tier": -1, 2024-09-15T20:43:22.803 INFO:tasks.workunit.client.0.smithi139.stdout: "write_tier": -1, 2024-09-15T20:43:22.803 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_mode": "none", 2024-09-15T20:43:22.803 INFO:tasks.workunit.client.0.smithi139.stdout: "target_max_bytes": 0, 2024-09-15T20:43:22.803 INFO:tasks.workunit.client.0.smithi139.stdout: "target_max_objects": 0, 2024-09-15T20:43:22.803 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T20:43:22.803 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T20:43:22.803 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T20:43:22.804 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_min_flush_age": 0, 2024-09-15T20:43:22.804 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_min_evict_age": 0, 2024-09-15T20:43:22.804 INFO:tasks.workunit.client.0.smithi139.stdout: "erasure_code_profile": "myprofile", 2024-09-15T20:43:22.804 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_params": { 2024-09-15T20:43:22.804 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "none" 2024-09-15T20:43:22.804 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.804 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_period": 0, 2024-09-15T20:43:22.804 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_count": 0, 2024-09-15T20:43:22.804 INFO:tasks.workunit.client.0.smithi139.stdout: "use_gmt_hitset": true, 2024-09-15T20:43:22.804 INFO:tasks.workunit.client.0.smithi139.stdout: "min_read_recency_for_promote": 0, 2024-09-15T20:43:22.804 INFO:tasks.workunit.client.0.smithi139.stdout: "min_write_recency_for_promote": 0, 2024-09-15T20:43:22.804 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T20:43:22.804 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_search_last_n": 0, 2024-09-15T20:43:22.804 INFO:tasks.workunit.client.0.smithi139.stdout: "grade_table": [], 2024-09-15T20:43:22.805 INFO:tasks.workunit.client.0.smithi139.stdout: "stripe_width": 8192, 2024-09-15T20:43:22.805 INFO:tasks.workunit.client.0.smithi139.stdout: "expected_num_objects": 0, 2024-09-15T20:43:22.805 INFO:tasks.workunit.client.0.smithi139.stdout: "fast_read": false, 2024-09-15T20:43:22.805 INFO:tasks.workunit.client.0.smithi139.stdout: "options": {}, 2024-09-15T20:43:22.805 INFO:tasks.workunit.client.0.smithi139.stdout: "application_metadata": {} 2024-09-15T20:43:22.805 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.805 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.805 INFO:tasks.workunit.client.0.smithi139.stdout: "osds": [ 2024-09-15T20:43:22.805 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.805 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:43:22.805 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "072ef209-0424-4dcb-ba88-59a19eaa46bd", 2024-09-15T20:43:22.805 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 1, 2024-09-15T20:43:22.805 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:43:22.805 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:43:22.805 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:43:22.806 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 5, 2024-09-15T20:43:22.806 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 29, 2024-09-15T20:43:22.806 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 37, 2024-09-15T20:43:22.806 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 42, 2024-09-15T20:43:22.806 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 31, 2024-09-15T20:43:22.806 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:43:22.806 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:43:22.806 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:43:22.806 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.806 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:43:22.806 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6802", 2024-09-15T20:43:22.806 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1240929079 2024-09-15T20:43:22.806 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.807 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.807 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:43:22.807 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6803", 2024-09-15T20:43:22.807 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1240929079 2024-09-15T20:43:22.807 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.807 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.807 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.807 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:43:22.807 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:43:22.807 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.807 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:43:22.807 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6804", 2024-09-15T20:43:22.807 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1240929079 2024-09-15T20:43:22.808 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.808 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.808 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:43:22.808 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6805", 2024-09-15T20:43:22.808 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1240929079 2024-09-15T20:43:22.808 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.808 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.808 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.808 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:43:22.808 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:43:22.808 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.808 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:43:22.809 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6808", 2024-09-15T20:43:22.809 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1240929079 2024-09-15T20:43:22.809 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.809 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.809 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:43:22.809 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6809", 2024-09-15T20:43:22.809 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1240929079 2024-09-15T20:43:22.809 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.809 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.809 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.809 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:43:22.809 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:43:22.809 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.810 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:43:22.810 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6806", 2024-09-15T20:43:22.810 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1240929079 2024-09-15T20:43:22.810 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.810 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.810 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:43:22.810 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6807", 2024-09-15T20:43:22.810 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 1240929079 2024-09-15T20:43:22.810 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.810 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.810 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.811 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6803/1240929079", 2024-09-15T20:43:22.811 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6805/1240929079", 2024-09-15T20:43:22.811 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1240929079", 2024-09-15T20:43:22.811 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1240929079", 2024-09-15T20:43:22.811 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:43:22.811 INFO:tasks.workunit.client.0.smithi139.stdout: "exists", 2024-09-15T20:43:22.811 INFO:tasks.workunit.client.0.smithi139.stdout: "up" 2024-09-15T20:43:22.811 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.811 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.811 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.811 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:43:22.811 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "61477bed-b55a-47e6-8ac3-ac4772590dce", 2024-09-15T20:43:22.811 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 0, 2024-09-15T20:43:22.811 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:43:22.812 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:43:22.812 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:43:22.812 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 0, 2024-09-15T20:43:22.812 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 0, 2024-09-15T20:43:22.812 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 10, 2024-09-15T20:43:22.812 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 25, 2024-09-15T20:43:22.812 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 32, 2024-09-15T20:43:22.812 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:43:22.812 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:43:22.812 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:43:22.812 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.812 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:43:22.812 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6810", 2024-09-15T20:43:22.813 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3833330082 2024-09-15T20:43:22.813 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.813 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.813 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:43:22.813 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6811", 2024-09-15T20:43:22.813 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3833330082 2024-09-15T20:43:22.813 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.813 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.813 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.813 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:43:22.814 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:43:22.814 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.814 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:43:22.814 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6812", 2024-09-15T20:43:22.814 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3833330082 2024-09-15T20:43:22.814 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.814 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.814 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:43:22.814 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6813", 2024-09-15T20:43:22.814 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3833330082 2024-09-15T20:43:22.814 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.814 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.814 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.814 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:43:22.815 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:43:22.815 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.815 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:43:22.815 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6816", 2024-09-15T20:43:22.815 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3833330082 2024-09-15T20:43:22.815 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.815 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.815 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:43:22.815 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6817", 2024-09-15T20:43:22.815 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3833330082 2024-09-15T20:43:22.815 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.815 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.816 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.816 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:43:22.816 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:43:22.816 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.816 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:43:22.816 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6814", 2024-09-15T20:43:22.816 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3833330082 2024-09-15T20:43:22.816 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.816 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.816 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:43:22.816 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6815", 2024-09-15T20:43:22.816 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3833330082 2024-09-15T20:43:22.816 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.816 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.817 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.817 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6811/3833330082", 2024-09-15T20:43:22.817 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6813/3833330082", 2024-09-15T20:43:22.817 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3833330082", 2024-09-15T20:43:22.817 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3833330082", 2024-09-15T20:43:22.817 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:43:22.817 INFO:tasks.workunit.client.0.smithi139.stdout: "exists" 2024-09-15T20:43:22.817 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.817 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.817 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.817 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:43:22.817 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "e0a289bd-3c1f-48e6-b612-9a2c75ec33bb", 2024-09-15T20:43:22.817 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 1, 2024-09-15T20:43:22.817 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:43:22.818 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:43:22.818 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:43:22.818 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 15, 2024-09-15T20:43:22.818 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 29, 2024-09-15T20:43:22.818 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 39, 2024-09-15T20:43:22.818 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 0, 2024-09-15T20:43:22.818 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 33, 2024-09-15T20:43:22.818 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:43:22.818 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:43:22.818 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:43:22.818 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.818 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:43:22.818 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6810", 2024-09-15T20:43:22.818 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 85082059 2024-09-15T20:43:22.819 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.819 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.819 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:43:22.819 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6811", 2024-09-15T20:43:22.819 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 85082059 2024-09-15T20:43:22.819 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.819 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.819 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.819 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:43:22.819 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:43:22.819 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.819 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:43:22.819 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6812", 2024-09-15T20:43:22.819 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 85082059 2024-09-15T20:43:22.820 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.820 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.820 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:43:22.820 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6813", 2024-09-15T20:43:22.820 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 85082059 2024-09-15T20:43:22.820 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.820 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.820 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.820 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:43:22.820 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:43:22.820 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.820 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:43:22.820 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6816", 2024-09-15T20:43:22.820 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 85082059 2024-09-15T20:43:22.821 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.821 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.821 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:43:22.821 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6817", 2024-09-15T20:43:22.821 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 85082059 2024-09-15T20:43:22.821 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.821 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.821 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.821 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:43:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:43:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:43:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6814", 2024-09-15T20:43:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 85082059 2024-09-15T20:43:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:43:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6815", 2024-09-15T20:43:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 85082059 2024-09-15T20:43:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.822 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6811/85082059", 2024-09-15T20:43:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6813/85082059", 2024-09-15T20:43:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6817/85082059", 2024-09-15T20:43:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6815/85082059", 2024-09-15T20:43:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:43:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: "exists", 2024-09-15T20:43:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: "up" 2024-09-15T20:43:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_xinfo": [ 2024-09-15T20:43:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:43:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:42:23.388013+0000", 2024-09-15T20:43:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:43:22.823 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:43:22.824 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:43:22.824 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:43:22.824 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:40:01.324476+0000", 2024-09-15T20:43:22.824 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:43:22.824 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.824 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.824 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:43:22.824 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:42:24.031556+0000", 2024-09-15T20:43:22.824 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:43:22.824 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:43:22.824 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:43:22.824 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:43:22.824 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:40:39.910012+0000", 2024-09-15T20:43:22.825 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:43:22.825 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.825 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.825 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:43:22.825 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:42:24.634134+0000", 2024-09-15T20:43:22.825 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:43:22.825 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:43:22.825 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:43:22.825 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:43:22.825 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:41:18.728508+0000", 2024-09-15T20:43:22.825 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:43:22.825 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.825 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.826 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap": [], 2024-09-15T20:43:22.826 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap_items": [], 2024-09-15T20:43:22.826 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap_primaries": [], 2024-09-15T20:43:22.826 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_temp": [], 2024-09-15T20:43:22.826 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_temp": [], 2024-09-15T20:43:22.826 INFO:tasks.workunit.client.0.smithi139.stdout: "blocklist": {}, 2024-09-15T20:43:22.826 INFO:tasks.workunit.client.0.smithi139.stdout: "range_blocklist": {}, 2024-09-15T20:43:22.826 INFO:tasks.workunit.client.0.smithi139.stdout: "erasure_code_profiles": { 2024-09-15T20:43:22.826 INFO:tasks.workunit.client.0.smithi139.stdout: "default": { 2024-09-15T20:43:22.826 INFO:tasks.workunit.client.0.smithi139.stdout: "k": "2", 2024-09-15T20:43:22.826 INFO:tasks.workunit.client.0.smithi139.stdout: "m": "2", 2024-09-15T20:43:22.826 INFO:tasks.workunit.client.0.smithi139.stdout: "plugin": "jerasure", 2024-09-15T20:43:22.826 INFO:tasks.workunit.client.0.smithi139.stdout: "technique": "reed_sol_van" 2024-09-15T20:43:22.826 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.827 INFO:tasks.workunit.client.0.smithi139.stdout: "myprofile": { 2024-09-15T20:43:22.827 INFO:tasks.workunit.client.0.smithi139.stdout: "crush-device-class": "", 2024-09-15T20:43:22.827 INFO:tasks.workunit.client.0.smithi139.stdout: "crush-failure-domain": "osd", 2024-09-15T20:43:22.827 INFO:tasks.workunit.client.0.smithi139.stdout: "crush-num-failure-domains": "0", 2024-09-15T20:43:22.827 INFO:tasks.workunit.client.0.smithi139.stdout: "crush-osds-per-failure-domain": "0", 2024-09-15T20:43:22.827 INFO:tasks.workunit.client.0.smithi139.stdout: "crush-root": "default", 2024-09-15T20:43:22.827 INFO:tasks.workunit.client.0.smithi139.stdout: "jerasure-per-chunk-alignment": "false", 2024-09-15T20:43:22.827 INFO:tasks.workunit.client.0.smithi139.stdout: "k": "2", 2024-09-15T20:43:22.827 INFO:tasks.workunit.client.0.smithi139.stdout: "m": "1", 2024-09-15T20:43:22.827 INFO:tasks.workunit.client.0.smithi139.stdout: "plugin": "jerasure", 2024-09-15T20:43:22.827 INFO:tasks.workunit.client.0.smithi139.stdout: "technique": "reed_sol_van", 2024-09-15T20:43:22.827 INFO:tasks.workunit.client.0.smithi139.stdout: "w": "8" 2024-09-15T20:43:22.827 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.827 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.827 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_snaps_queue": [], 2024-09-15T20:43:22.828 INFO:tasks.workunit.client.0.smithi139.stdout: "new_removed_snaps": [], 2024-09-15T20:43:22.828 INFO:tasks.workunit.client.0.smithi139.stdout: "new_purged_snaps": [], 2024-09-15T20:43:22.828 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_node_flags": {}, 2024-09-15T20:43:22.828 INFO:tasks.workunit.client.0.smithi139.stdout: "device_class_flags": {}, 2024-09-15T20:43:22.828 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode": { 2024-09-15T20:43:22.828 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode_enabled": false, 2024-09-15T20:43:22.828 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_bucket_count": 0, 2024-09-15T20:43:22.828 INFO:tasks.workunit.client.0.smithi139.stdout: "degraded_stretch_mode": 0, 2024-09-15T20:43:22.828 INFO:tasks.workunit.client.0.smithi139.stdout: "recovering_stretch_mode": 0, 2024-09-15T20:43:22.828 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode_bucket": 0 2024-09-15T20:43:22.828 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.828 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.828 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_metadata": [ 2024-09-15T20:43:22.828 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.829 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:43:22.829 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:43:22.829 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6804/1240929079,v1:127.0.0.1:6805/1240929079]", 2024-09-15T20:43:22.829 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:43:22.829 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:43:22.829 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:43:22.829 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:43:22.829 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:43:22.829 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:43:22.829 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:43:22.829 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:43:22.829 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:43:22.829 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:43:22.829 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:43:22.829 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-15T20:43:22.830 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:43:22.830 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:43:22.830 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:43:22.830 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:43:22.830 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:43:22.830 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:43:22.830 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:43:22.830 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:43:22.830 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:43:22.830 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:43:22.830 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:39:32.056892Z", 2024-09-15T20:43:22.830 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:43:22.830 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:43:22.830 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:43:22.831 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:43:22.831 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:43:22.831 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:43:22.831 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:43:22.831 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6802/1240929079,v1:127.0.0.1:6803/1240929079]", 2024-09-15T20:43:22.831 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:43:22.831 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1240929079,v1:127.0.0.1:6809/1240929079]", 2024-09-15T20:43:22.831 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1240929079,v1:127.0.0.1:6807/1240929079]", 2024-09-15T20:43:22.831 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:43:22.831 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:43:22.832 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:43:22.832 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:43:22.832 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:43:22.832 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:43:22.832 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:43:22.832 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:43:22.832 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:43:22.832 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/0", 2024-09-15T20:43:22.832 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:43:22.832 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:43:22.832 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:43:22.832 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.832 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.833 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:43:22.833 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:43:22.833 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6812/3833330082,v1:127.0.0.1:6813/3833330082]", 2024-09-15T20:43:22.833 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:43:22.833 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:43:22.833 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:43:22.833 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:43:22.833 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:43:22.833 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:43:22.833 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:43:22.834 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:43:22.834 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:43:22.834 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:43:22.834 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:43:22.834 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-15T20:43:22.834 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:43:22.834 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:43:22.834 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:43:22.834 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:43:22.834 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:43:22.835 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:43:22.835 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:43:22.835 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:43:22.835 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:43:22.835 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:43:22.835 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:40:08.814496Z", 2024-09-15T20:43:22.835 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:43:22.835 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:43:22.835 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:43:22.835 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:43:22.835 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:43:22.835 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:43:22.836 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:43:22.836 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6810/3833330082,v1:127.0.0.1:6811/3833330082]", 2024-09-15T20:43:22.836 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:43:22.836 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3833330082,v1:127.0.0.1:6817/3833330082]", 2024-09-15T20:43:22.836 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3833330082,v1:127.0.0.1:6815/3833330082]", 2024-09-15T20:43:22.836 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:43:22.836 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:43:22.836 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:43:22.836 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:43:22.836 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:43:22.836 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:43:22.836 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:43:22.837 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:43:22.837 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:43:22.837 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/1", 2024-09-15T20:43:22.837 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:43:22.837 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:43:22.837 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:43:22.837 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.837 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.837 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:43:22.837 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:43:22.837 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6812/85082059,v1:127.0.0.1:6813/85082059]", 2024-09-15T20:43:22.837 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:43:22.837 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:43:22.838 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:43:22.838 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:43:22.838 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:43:22.838 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:43:22.838 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:43:22.838 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:43:22.838 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:43:22.838 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:43:22.838 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:43:22.838 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-15T20:43:22.838 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:43:22.838 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:43:22.838 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:43:22.839 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:43:22.839 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:43:22.839 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:43:22.839 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:43:22.839 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:43:22.839 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:43:22.839 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:43:22.839 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:40:48.254151Z", 2024-09-15T20:43:22.839 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:43:22.839 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:43:22.839 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:43:22.839 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:43:22.839 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:43:22.840 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:43:22.840 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:43:22.840 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6810/85082059,v1:127.0.0.1:6811/85082059]", 2024-09-15T20:43:22.840 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:43:22.840 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/85082059,v1:127.0.0.1:6817/85082059]", 2024-09-15T20:43:22.840 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/85082059,v1:127.0.0.1:6815/85082059]", 2024-09-15T20:43:22.840 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:43:22.840 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:43:22.840 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:43:22.840 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:43:22.840 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:43:22.840 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:43:22.840 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:43:22.840 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:43:22.840 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:43:22.841 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/2", 2024-09-15T20:43:22.841 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:43:22.841 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:43:22.841 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:43:22.841 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.841 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.841 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_clean_epochs": { 2024-09-15T20:43:22.841 INFO:tasks.workunit.client.0.smithi139.stdout: "min_last_epoch_clean": 0, 2024-09-15T20:43:22.841 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_clean": { 2024-09-15T20:43:22.841 INFO:tasks.workunit.client.0.smithi139.stdout: "per_pool": [] 2024-09-15T20:43:22.841 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.841 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_epochs": [ 2024-09-15T20:43:22.841 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.841 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:43:22.841 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 37 2024-09-15T20:43:22.842 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.842 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.842 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:43:22.842 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 39 2024-09-15T20:43:22.842 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.842 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.842 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.842 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_first_committed": 1, 2024-09-15T20:43:22.842 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_last_committed": 43, 2024-09-15T20:43:22.842 INFO:tasks.workunit.client.0.smithi139.stdout: "crushmap": { 2024-09-15T20:43:22.842 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": [ 2024-09-15T20:43:22.842 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.842 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:43:22.842 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.0", 2024-09-15T20:43:22.843 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:43:22.843 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.843 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.843 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:43:22.843 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.1", 2024-09-15T20:43:22.843 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:43:22.843 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.843 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.843 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:43:22.843 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.2", 2024-09-15T20:43:22.843 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:43:22.843 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.843 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.843 INFO:tasks.workunit.client.0.smithi139.stdout: "types": [ 2024-09-15T20:43:22.843 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.844 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 0, 2024-09-15T20:43:22.844 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd" 2024-09-15T20:43:22.844 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.844 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.844 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:43:22.844 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "host" 2024-09-15T20:43:22.844 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.844 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.844 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 2, 2024-09-15T20:43:22.844 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "chassis" 2024-09-15T20:43:22.844 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.844 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.844 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 3, 2024-09-15T20:43:22.844 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "rack" 2024-09-15T20:43:22.845 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.845 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.845 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 4, 2024-09-15T20:43:22.845 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "row" 2024-09-15T20:43:22.845 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.845 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.845 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 5, 2024-09-15T20:43:22.845 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "pdu" 2024-09-15T20:43:22.845 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.845 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.845 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 6, 2024-09-15T20:43:22.845 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "pod" 2024-09-15T20:43:22.845 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.845 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.845 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 7, 2024-09-15T20:43:22.846 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "room" 2024-09-15T20:43:22.846 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.846 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.846 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 8, 2024-09-15T20:43:22.846 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "datacenter" 2024-09-15T20:43:22.846 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.846 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.846 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 9, 2024-09-15T20:43:22.846 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "zone" 2024-09-15T20:43:22.846 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.846 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.846 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 10, 2024-09-15T20:43:22.846 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "region" 2024-09-15T20:43:22.846 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.846 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.847 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:43:22.847 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "root" 2024-09-15T20:43:22.847 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.847 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.847 INFO:tasks.workunit.client.0.smithi139.stdout: "buckets": [ 2024-09-15T20:43:22.847 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.847 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -1, 2024-09-15T20:43:22.847 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "default", 2024-09-15T20:43:22.847 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:43:22.847 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "root", 2024-09-15T20:43:22.847 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:43:22.847 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:43:22.847 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:43:22.847 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:43:22.848 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.848 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -3, 2024-09-15T20:43:22.848 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:43:22.848 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:43:22.848 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.848 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.848 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.848 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.848 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -2, 2024-09-15T20:43:22.848 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "default~hdd", 2024-09-15T20:43:22.848 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:43:22.848 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "root", 2024-09-15T20:43:22.848 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:43:22.848 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:43:22.849 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:43:22.849 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:43:22.849 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.849 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -4, 2024-09-15T20:43:22.849 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:43:22.849 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:43:22.849 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.849 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.849 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.849 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.849 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -3, 2024-09-15T20:43:22.849 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "smithi139", 2024-09-15T20:43:22.849 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:43:22.849 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "host", 2024-09-15T20:43:22.850 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:43:22.850 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:43:22.850 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:43:22.850 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:43:22.850 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.850 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:43:22.850 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:43:22.850 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:43:22.850 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.850 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.850 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:43:22.850 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:43:22.850 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 1 2024-09-15T20:43:22.850 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.850 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.851 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:43:22.851 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:43:22.851 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 2 2024-09-15T20:43:22.851 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.851 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.851 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.851 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.851 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -4, 2024-09-15T20:43:22.851 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "smithi139~hdd", 2024-09-15T20:43:22.851 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:43:22.851 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "host", 2024-09-15T20:43:22.851 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:43:22.851 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:43:22.851 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:43:22.852 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:43:22.852 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.852 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:43:22.852 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:43:22.852 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:43:22.852 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.852 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.852 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:43:22.852 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:43:22.852 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 1 2024-09-15T20:43:22.852 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.852 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.852 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:43:22.852 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:43:22.852 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 2 2024-09-15T20:43:22.853 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.853 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.853 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.853 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.853 INFO:tasks.workunit.client.0.smithi139.stdout: "rules": [ 2024-09-15T20:43:22.853 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.853 INFO:tasks.workunit.client.0.smithi139.stdout: "rule_id": 0, 2024-09-15T20:43:22.853 INFO:tasks.workunit.client.0.smithi139.stdout: "rule_name": "replicated_rule", 2024-09-15T20:43:22.853 INFO:tasks.workunit.client.0.smithi139.stdout: "type": 1, 2024-09-15T20:43:22.853 INFO:tasks.workunit.client.0.smithi139.stdout: "steps": [ 2024-09-15T20:43:22.853 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.853 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "take", 2024-09-15T20:43:22.853 INFO:tasks.workunit.client.0.smithi139.stdout: "item": -1, 2024-09-15T20:43:22.853 INFO:tasks.workunit.client.0.smithi139.stdout: "item_name": "default" 2024-09-15T20:43:22.854 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.854 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.854 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "choose_firstn", 2024-09-15T20:43:22.854 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 0, 2024-09-15T20:43:22.854 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "osd" 2024-09-15T20:43:22.854 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.854 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.854 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "emit" 2024-09-15T20:43:22.854 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.854 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.854 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.854 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.855 INFO:tasks.workunit.client.0.smithi139.stdout: "rule_id": 1, 2024-09-15T20:43:22.855 INFO:tasks.workunit.client.0.smithi139.stdout: "rule_name": "test", 2024-09-15T20:43:22.855 INFO:tasks.workunit.client.0.smithi139.stdout: "type": 3, 2024-09-15T20:43:22.855 INFO:tasks.workunit.client.0.smithi139.stdout: "steps": [ 2024-09-15T20:43:22.855 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.855 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "set_chooseleaf_tries", 2024-09-15T20:43:22.855 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 5 2024-09-15T20:43:22.855 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.855 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.855 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "set_choose_tries", 2024-09-15T20:43:22.855 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 100 2024-09-15T20:43:22.855 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.855 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.856 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "take", 2024-09-15T20:43:22.856 INFO:tasks.workunit.client.0.smithi139.stdout: "item": -1, 2024-09-15T20:43:22.856 INFO:tasks.workunit.client.0.smithi139.stdout: "item_name": "default" 2024-09-15T20:43:22.856 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.856 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.856 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "choose_indep", 2024-09-15T20:43:22.856 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 0, 2024-09-15T20:43:22.856 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "osd" 2024-09-15T20:43:22.856 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.856 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.856 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "emit" 2024-09-15T20:43:22.856 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.856 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:43:22.856 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.856 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.857 INFO:tasks.workunit.client.0.smithi139.stdout: "tunables": { 2024-09-15T20:43:22.857 INFO:tasks.workunit.client.0.smithi139.stderr:report 3679299248 2024-09-15T20:43:22.857 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_local_tries": 0, 2024-09-15T20:43:22.857 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_local_fallback_tries": 0, 2024-09-15T20:43:22.857 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_total_tries": 50, 2024-09-15T20:43:22.858 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_descend_once": 1, 2024-09-15T20:43:22.858 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_vary_r": 1, 2024-09-15T20:43:22.858 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_stable": 1, 2024-09-15T20:43:22.858 INFO:tasks.workunit.client.0.smithi139.stdout: "msr_descents": 100, 2024-09-15T20:43:22.858 INFO:tasks.workunit.client.0.smithi139.stdout: "msr_collision_tries": 100, 2024-09-15T20:43:22.858 INFO:tasks.workunit.client.0.smithi139.stdout: "straw_calc_version": 1, 2024-09-15T20:43:22.858 INFO:tasks.workunit.client.0.smithi139.stdout: "allowed_bucket_algs": 54, 2024-09-15T20:43:22.858 INFO:tasks.workunit.client.0.smithi139.stdout: "profile": "jewel", 2024-09-15T20:43:22.858 INFO:tasks.workunit.client.0.smithi139.stdout: "optimal_tunables": 1, 2024-09-15T20:43:22.858 INFO:tasks.workunit.client.0.smithi139.stdout: "legacy_tunables": 0, 2024-09-15T20:43:22.858 INFO:tasks.workunit.client.0.smithi139.stdout: "minimum_required_version": "jewel", 2024-09-15T20:43:22.858 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables": 1, 2024-09-15T20:43:22.858 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables2": 1, 2024-09-15T20:43:22.858 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v2_rules": 1, 2024-09-15T20:43:22.858 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables3": 1, 2024-09-15T20:43:22.858 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v3_rules": 0, 2024-09-15T20:43:22.859 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v4_buckets": 1, 2024-09-15T20:43:22.859 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables5": 1, 2024-09-15T20:43:22.859 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v5_rules": 0, 2024-09-15T20:43:22.859 INFO:tasks.workunit.client.0.smithi139.stdout: "has_msr_rules": 0 2024-09-15T20:43:22.859 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.859 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_args": {} 2024-09-15T20:43:22.859 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.859 INFO:tasks.workunit.client.0.smithi139.stdout: "fsmap": { 2024-09-15T20:43:22.859 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 1, 2024-09-15T20:43:22.859 INFO:tasks.workunit.client.0.smithi139.stdout: "btime": "2024-09-15T20:39:25:184970+0000", 2024-09-15T20:43:22.859 INFO:tasks.workunit.client.0.smithi139.stdout: "default_fscid": -1, 2024-09-15T20:43:22.859 INFO:tasks.workunit.client.0.smithi139.stdout: "compat": { 2024-09-15T20:43:22.859 INFO:tasks.workunit.client.0.smithi139.stdout: "compat": {}, 2024-09-15T20:43:22.860 INFO:tasks.workunit.client.0.smithi139.stdout: "ro_compat": {}, 2024-09-15T20:43:22.860 INFO:tasks.workunit.client.0.smithi139.stdout: "incompat": { 2024-09-15T20:43:22.860 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_1": "base v0.20", 2024-09-15T20:43:22.860 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_2": "client writeable ranges", 2024-09-15T20:43:22.860 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_3": "default file layouts on dirs", 2024-09-15T20:43:22.860 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_4": "dir inode in separate object", 2024-09-15T20:43:22.860 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_5": "mds uses versioned encoding", 2024-09-15T20:43:22.860 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-15T20:43:22.860 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_8": "no anchor table", 2024-09-15T20:43:22.860 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_9": "file layout v2", 2024-09-15T20:43:22.860 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_10": "snaprealm v2", 2024-09-15T20:43:22.861 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_11": "minor log segments", 2024-09-15T20:43:22.861 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_12": "quiesce subvolumes" 2024-09-15T20:43:22.861 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.861 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.861 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_flags": { 2024-09-15T20:43:22.861 INFO:tasks.workunit.client.0.smithi139.stdout: "enable_multiple": true, 2024-09-15T20:43:22.861 INFO:tasks.workunit.client.0.smithi139.stdout: "ever_enabled_multiple": true 2024-09-15T20:43:22.861 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.861 INFO:tasks.workunit.client.0.smithi139.stdout: "standbys": [], 2024-09-15T20:43:22.861 INFO:tasks.workunit.client.0.smithi139.stdout: "filesystems": [] 2024-09-15T20:43:22.861 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.861 INFO:tasks.workunit.client.0.smithi139.stdout: "mdsmap_first_committed": 1, 2024-09-15T20:43:22.861 INFO:tasks.workunit.client.0.smithi139.stdout: "mdsmap_last_committed": 1, 2024-09-15T20:43:22.862 INFO:tasks.workunit.client.0.smithi139.stdout: "auth": { 2024-09-15T20:43:22.862 INFO:tasks.workunit.client.0.smithi139.stdout: "first_committed": 1, 2024-09-15T20:43:22.862 INFO:tasks.workunit.client.0.smithi139.stdout: "last_committed": 5, 2024-09-15T20:43:22.862 INFO:tasks.workunit.client.0.smithi139.stdout: "num_secrets": 3 2024-09-15T20:43:22.862 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.862 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg": 1, 2024-09-15T20:43:22.863 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_active": 1, 2024-09-15T20:43:22.864 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_unknown": 0, 2024-09-15T20:43:22.864 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osd": 3, 2024-09-15T20:43:22.864 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_sum": { 2024-09-15T20:43:22.864 INFO:tasks.workunit.client.0.smithi139.stdout: "stat_sum": { 2024-09-15T20:43:22.864 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes": 62667, 2024-09-15T20:43:22.864 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects": 101, 2024-09-15T20:43:22.864 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_clones": 1, 2024-09-15T20:43:22.864 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_copies": 303, 2024-09-15T20:43:22.864 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T20:43:22.864 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing": 0, 2024-09-15T20:43:22.864 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_degraded": 101, 2024-09-15T20:43:22.865 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_misplaced": 0, 2024-09-15T20:43:22.865 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_unfound": 0, 2024-09-15T20:43:22.865 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_dirty": 101, 2024-09-15T20:43:22.865 INFO:tasks.workunit.client.0.smithi139.stdout: "num_whiteouts": 0, 2024-09-15T20:43:22.865 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read": 0, 2024-09-15T20:43:22.865 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read_kb": 0, 2024-09-15T20:43:22.865 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write": 101, 2024-09-15T20:43:22.865 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write_kb": 101, 2024-09-15T20:43:22.865 INFO:tasks.workunit.client.0.smithi139.stdout: "num_scrub_errors": 0, 2024-09-15T20:43:22.865 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T20:43:22.865 INFO:tasks.workunit.client.0.smithi139.stdout: "num_deep_scrub_errors": 0, 2024-09-15T20:43:22.865 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_recovered": 0, 2024-09-15T20:43:22.865 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_recovered": 0, 2024-09-15T20:43:22.865 INFO:tasks.workunit.client.0.smithi139.stdout: "num_keys_recovered": 0, 2024-09-15T20:43:22.865 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_omap": 0, 2024-09-15T20:43:22.866 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T20:43:22.866 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T20:43:22.866 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush": 0, 2024-09-15T20:43:22.866 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_kb": 0, 2024-09-15T20:43:22.866 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict": 0, 2024-09-15T20:43:22.866 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_kb": 0, 2024-09-15T20:43:22.866 INFO:tasks.workunit.client.0.smithi139.stdout: "num_promote": 0, 2024-09-15T20:43:22.866 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_high": 0, 2024-09-15T20:43:22.866 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_low": 0, 2024-09-15T20:43:22.866 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_some": 0, 2024-09-15T20:43:22.866 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_full": 0, 2024-09-15T20:43:22.866 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_pinned": 0, 2024-09-15T20:43:22.866 INFO:tasks.workunit.client.0.smithi139.stdout: "num_legacy_snapsets": 0, 2024-09-15T20:43:22.866 INFO:tasks.workunit.client.0.smithi139.stdout: "num_large_omap_objects": 0, 2024-09-15T20:43:22.867 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_manifest": 0, 2024-09-15T20:43:22.867 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_bytes": 0, 2024-09-15T20:43:22.867 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_keys": 0, 2024-09-15T20:43:22.867 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_repaired": 0 2024-09-15T20:43:22.867 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.867 INFO:tasks.workunit.client.0.smithi139.stdout: "store_stats": { 2024-09-15T20:43:22.867 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 0, 2024-09-15T20:43:22.867 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 0, 2024-09-15T20:43:22.867 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:43:22.867 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 0, 2024-09-15T20:43:22.867 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 0, 2024-09-15T20:43:22.867 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:43:22.867 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:43:22.867 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:43:22.868 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 0, 2024-09-15T20:43:22.868 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 0 2024-09-15T20:43:22.868 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.868 INFO:tasks.workunit.client.0.smithi139.stdout: "log_size": 18, 2024-09-15T20:43:22.868 INFO:tasks.workunit.client.0.smithi139.stdout: "ondisk_log_size": 18, 2024-09-15T20:43:22.868 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 3, 2024-09-15T20:43:22.868 INFO:tasks.workunit.client.0.smithi139.stdout: "acting": 3, 2024-09-15T20:43:22.868 INFO:tasks.workunit.client.0.smithi139.stdout: "num_store_stats": 0 2024-09-15T20:43:22.868 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.868 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_sum": { 2024-09-15T20:43:22.868 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 0, 2024-09-15T20:43:22.868 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 0, 2024-09-15T20:43:22.868 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pgs": 3, 2024-09-15T20:43:22.868 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osds": 3, 2024-09-15T20:43:22.868 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_osds": 3, 2024-09-15T20:43:22.869 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T20:43:22.869 INFO:tasks.workunit.client.0.smithi139.stdout: "kb": 314572800, 2024-09-15T20:43:22.869 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used": 83696, 2024-09-15T20:43:22.869 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_data": 2164, 2024-09-15T20:43:22.869 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_omap": 10, 2024-09-15T20:43:22.869 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_meta": 81461, 2024-09-15T20:43:22.869 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_avail": 314489104, 2024-09-15T20:43:22.869 INFO:tasks.workunit.client.0.smithi139.stdout: "statfs": { 2024-09-15T20:43:22.869 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 322122547200, 2024-09-15T20:43:22.869 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 322036842496, 2024-09-15T20:43:22.869 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:43:22.869 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 2215936, 2024-09-15T20:43:22.869 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 1550701, 2024-09-15T20:43:22.869 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:43:22.869 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:43:22.870 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:43:22.870 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 10396, 2024-09-15T20:43:22.870 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 83416932 2024-09-15T20:43:22.870 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.870 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_peers": [], 2024-09-15T20:43:22.870 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_trim_queue_len": 0, 2024-09-15T20:43:22.870 INFO:tasks.workunit.client.0.smithi139.stdout: "num_snap_trimming": 0, 2024-09-15T20:43:22.870 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shards_repaired": 0, 2024-09-15T20:43:22.870 INFO:tasks.workunit.client.0.smithi139.stdout: "op_queue_age_hist": { 2024-09-15T20:43:22.870 INFO:tasks.workunit.client.0.smithi139.stdout: "histogram": [], 2024-09-15T20:43:22.870 INFO:tasks.workunit.client.0.smithi139.stdout: "upper_bound": 1 2024-09-15T20:43:22.870 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.870 INFO:tasks.workunit.client.0.smithi139.stdout: "perf_stat": { 2024-09-15T20:43:22.870 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ms": 0, 2024-09-15T20:43:22.870 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ms": 0, 2024-09-15T20:43:22.871 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ns": 0, 2024-09-15T20:43:22.871 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ns": 0 2024-09-15T20:43:22.871 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.871 INFO:tasks.workunit.client.0.smithi139.stdout: "alerts": [], 2024-09-15T20:43:22.871 INFO:tasks.workunit.client.0.smithi139.stdout: "network_ping_times": [] 2024-09-15T20:43:22.871 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.871 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_sum_by_class": { 2024-09-15T20:43:22.871 INFO:tasks.workunit.client.0.smithi139.stdout: "hdd": { 2024-09-15T20:43:22.871 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 0, 2024-09-15T20:43:22.871 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 0, 2024-09-15T20:43:22.871 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pgs": 3, 2024-09-15T20:43:22.871 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osds": 3, 2024-09-15T20:43:22.871 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_osds": 3, 2024-09-15T20:43:22.871 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T20:43:22.871 INFO:tasks.workunit.client.0.smithi139.stdout: "kb": 314572800, 2024-09-15T20:43:22.872 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used": 83696, 2024-09-15T20:43:22.872 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_data": 2164, 2024-09-15T20:43:22.872 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_omap": 10, 2024-09-15T20:43:22.872 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_meta": 81461, 2024-09-15T20:43:22.872 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_avail": 314489104, 2024-09-15T20:43:22.872 INFO:tasks.workunit.client.0.smithi139.stdout: "statfs": { 2024-09-15T20:43:22.872 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 322122547200, 2024-09-15T20:43:22.872 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 322036842496, 2024-09-15T20:43:22.872 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:43:22.872 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 2215936, 2024-09-15T20:43:22.872 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 1550701, 2024-09-15T20:43:22.872 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:43:22.872 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:43:22.872 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:43:22.873 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 10396, 2024-09-15T20:43:22.873 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 83416932 2024-09-15T20:43:22.873 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.873 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_peers": [], 2024-09-15T20:43:22.873 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_trim_queue_len": 0, 2024-09-15T20:43:22.873 INFO:tasks.workunit.client.0.smithi139.stdout: "num_snap_trimming": 0, 2024-09-15T20:43:22.873 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shards_repaired": 0, 2024-09-15T20:43:22.873 INFO:tasks.workunit.client.0.smithi139.stdout: "op_queue_age_hist": { 2024-09-15T20:43:22.873 INFO:tasks.workunit.client.0.smithi139.stdout: "histogram": [], 2024-09-15T20:43:22.873 INFO:tasks.workunit.client.0.smithi139.stdout: "upper_bound": 1 2024-09-15T20:43:22.873 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.873 INFO:tasks.workunit.client.0.smithi139.stdout: "perf_stat": { 2024-09-15T20:43:22.873 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ms": 0, 2024-09-15T20:43:22.874 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ms": 0, 2024-09-15T20:43:22.874 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ns": 0, 2024-09-15T20:43:22.874 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ns": 0 2024-09-15T20:43:22.874 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.874 INFO:tasks.workunit.client.0.smithi139.stdout: "alerts": [], 2024-09-15T20:43:22.874 INFO:tasks.workunit.client.0.smithi139.stdout: "network_ping_times": [] 2024-09-15T20:43:22.874 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.874 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.874 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_stats": [ 2024-09-15T20:43:22.874 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.874 INFO:tasks.workunit.client.0.smithi139.stdout: "poolid": 1, 2024-09-15T20:43:22.874 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg": 1, 2024-09-15T20:43:22.874 INFO:tasks.workunit.client.0.smithi139.stdout: "stat_sum": { 2024-09-15T20:43:22.874 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes": 62667, 2024-09-15T20:43:22.875 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects": 101, 2024-09-15T20:43:22.875 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_clones": 1, 2024-09-15T20:43:22.875 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_copies": 303, 2024-09-15T20:43:22.875 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T20:43:22.875 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing": 0, 2024-09-15T20:43:22.875 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_degraded": 101, 2024-09-15T20:43:22.875 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_misplaced": 0, 2024-09-15T20:43:22.875 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_unfound": 0, 2024-09-15T20:43:22.875 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_dirty": 101, 2024-09-15T20:43:22.875 INFO:tasks.workunit.client.0.smithi139.stdout: "num_whiteouts": 0, 2024-09-15T20:43:22.875 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read": 0, 2024-09-15T20:43:22.875 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read_kb": 0, 2024-09-15T20:43:22.875 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write": 101, 2024-09-15T20:43:22.875 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write_kb": 101, 2024-09-15T20:43:22.875 INFO:tasks.workunit.client.0.smithi139.stdout: "num_scrub_errors": 0, 2024-09-15T20:43:22.876 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T20:43:22.876 INFO:tasks.workunit.client.0.smithi139.stdout: "num_deep_scrub_errors": 0, 2024-09-15T20:43:22.876 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_recovered": 0, 2024-09-15T20:43:22.876 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_recovered": 0, 2024-09-15T20:43:22.876 INFO:tasks.workunit.client.0.smithi139.stdout: "num_keys_recovered": 0, 2024-09-15T20:43:22.876 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_omap": 0, 2024-09-15T20:43:22.876 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T20:43:22.876 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T20:43:22.876 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush": 0, 2024-09-15T20:43:22.876 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_kb": 0, 2024-09-15T20:43:22.876 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict": 0, 2024-09-15T20:43:22.876 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_kb": 0, 2024-09-15T20:43:22.876 INFO:tasks.workunit.client.0.smithi139.stdout: "num_promote": 0, 2024-09-15T20:43:22.877 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_high": 0, 2024-09-15T20:43:22.877 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_low": 0, 2024-09-15T20:43:22.877 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_some": 0, 2024-09-15T20:43:22.877 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_full": 0, 2024-09-15T20:43:22.877 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_pinned": 0, 2024-09-15T20:43:22.877 INFO:tasks.workunit.client.0.smithi139.stdout: "num_legacy_snapsets": 0, 2024-09-15T20:43:22.877 INFO:tasks.workunit.client.0.smithi139.stdout: "num_large_omap_objects": 0, 2024-09-15T20:43:22.877 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_manifest": 0, 2024-09-15T20:43:22.877 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_bytes": 0, 2024-09-15T20:43:22.877 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_keys": 0, 2024-09-15T20:43:22.877 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_repaired": 0 2024-09-15T20:43:22.877 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.877 INFO:tasks.workunit.client.0.smithi139.stdout: "store_stats": { 2024-09-15T20:43:22.878 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 0, 2024-09-15T20:43:22.878 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 0, 2024-09-15T20:43:22.878 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:43:22.878 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 1241088, 2024-09-15T20:43:22.878 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 1241088, 2024-09-15T20:43:22.878 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:43:22.878 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:43:22.878 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:43:22.878 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 0, 2024-09-15T20:43:22.878 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 0 2024-09-15T20:43:22.878 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.878 INFO:tasks.workunit.client.0.smithi139.stdout: "log_size": 18, 2024-09-15T20:43:22.878 INFO:tasks.workunit.client.0.smithi139.stdout: "ondisk_log_size": 18, 2024-09-15T20:43:22.878 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 3, 2024-09-15T20:43:22.879 INFO:tasks.workunit.client.0.smithi139.stdout: "acting": 3, 2024-09-15T20:43:22.879 INFO:tasks.workunit.client.0.smithi139.stdout: "num_store_stats": 3 2024-09-15T20:43:22.879 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.879 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.879 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_stats": [ 2024-09-15T20:43:22.879 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.879 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:43:22.879 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 158913789964 2024-09-15T20:43:22.879 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.879 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.879 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:43:22.879 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 42949672993 2024-09-15T20:43:22.879 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.879 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.880 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:43:22.880 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 167503724555 2024-09-15T20:43:22.880 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.880 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.880 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_by_state": [ 2024-09-15T20:43:22.880 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.880 INFO:tasks.workunit.client.0.smithi139.stdout: "state": "active+undersized+degraded", 2024-09-15T20:43:22.880 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 1 2024-09-15T20:43:22.880 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.880 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.880 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_by_osd": [ 2024-09-15T20:43:22.880 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.880 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2147483647, 2024-09-15T20:43:22.880 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 0, 2024-09-15T20:43:22.881 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 1, 2024-09-15T20:43:22.881 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:43:22.881 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.881 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.881 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:43:22.881 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 0, 2024-09-15T20:43:22.881 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 1, 2024-09-15T20:43:22.881 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:43:22.881 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.881 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.881 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:43:22.881 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 1, 2024-09-15T20:43:22.881 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 1, 2024-09-15T20:43:22.882 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:43:22.882 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.882 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.882 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:43:22.882 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 0, 2024-09-15T20:43:22.882 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 0, 2024-09-15T20:43:22.882 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:43:22.882 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.882 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.882 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snaps": [ 2024-09-15T20:43:22.882 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:43:22.882 INFO:tasks.workunit.client.0.smithi139.stdout: "pool": 1, 2024-09-15T20:43:22.882 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snaps": {} 2024-09-15T20:43:22.882 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.883 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:43:22.883 INFO:tasks.workunit.client.0.smithi139.stdout: "servicemap": { 2024-09-15T20:43:22.883 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 5, 2024-09-15T20:43:22.883 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:42:20.827915+0000", 2024-09-15T20:43:22.883 INFO:tasks.workunit.client.0.smithi139.stdout: "services": {} 2024-09-15T20:43:22.883 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:43:22.883 INFO:tasks.workunit.client.0.smithi139.stdout: "mgrstat_first_committed": 1, 2024-09-15T20:43:22.883 INFO:tasks.workunit.client.0.smithi139.stdout: "mgrstat_last_committed": 107, 2024-09-15T20:43:22.883 INFO:tasks.workunit.client.0.smithi139.stdout: "logm_first_committed": 1, 2024-09-15T20:43:22.883 INFO:tasks.workunit.client.0.smithi139.stdout: "logm_last_committed": 418, 2024-09-15T20:43:22.883 INFO:tasks.workunit.client.0.smithi139.stdout: "paxos": { 2024-09-15T20:43:22.883 INFO:tasks.workunit.client.0.smithi139.stdout: "first_committed": 1, 2024-09-15T20:43:22.883 INFO:tasks.workunit.client.0.smithi139.stdout: "last_committed": 578, 2024-09-15T20:43:22.883 INFO:tasks.workunit.client.0.smithi139.stdout: "last_pn": 0, 2024-09-15T20:43:22.883 INFO:tasks.workunit.client.0.smithi139.stdout: "accepted_pn": 0 2024-09-15T20:43:22.884 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:43:22.884 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:43:22.884 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-15T20:43:22.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-09-15T20:43:22.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2024-09-15T20:43:22.885 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:43:22.885 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:43:22.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:43:22.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:43:22.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:43:22.911 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:43:22.911 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2024-09-15T20:43:22.911 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-15T20:43:22.911 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:43:22.911 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-15T20:43:22.911 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:43:22.911 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-09-15T20:43:22.911 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2024-09-15T20:43:24.476 INFO:tasks.workunit.client.0.smithi139.stderr:PG '1.0' not found 2024-09-15T20:43:24.775 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2024-09-15T20:43:24.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-09-15T20:43:24.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-15T20:43:24.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:43:24.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:43:24.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:43:24.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-15T20:43:24.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:43:24.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:43:24.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:43:24.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:43:24.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-15T20:43:24.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-15T20:43:24.777 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:43:24.777 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:43:24.777 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:43:24.777 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:43:24.777 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:43:24.777 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:43:24.778 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:43:24.778 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:43:24.778 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:43:24.778 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:43:24.778 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:43:24.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:43:24.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:43:24.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:43:24.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:43:24.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:43:24.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:43:24.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:43:24.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:43:24.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:43:24.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-15T20:43:24.780 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T20:43:24.780 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:43:24.780 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:43:24.780 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-15T20:43:24.781 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:43:24.782 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:43:24.782 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:43:24.782 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:43:24.822 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:43:24.826+0000 7f355aba07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:43:24.825 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:43:24.830+0000 7f355aba07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:43:24.826 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:43:24.830+0000 7f355aba07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:43:25.278 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:43:25.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:43:25.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:43:25.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:43:25.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:43:25.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:25.279 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:43:25.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:43:25.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:25.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:43:25.761 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 37 up_thru 42 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/1240929079,v1:127.0.0.1:6803/1240929079] [v2:127.0.0.1:6804/1240929079,v1:127.0.0.1:6805/1240929079] exists,up 072ef209-0424-4dcb-ba88-59a19eaa46bd 2024-09-15T20:43:25.762 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:43:25.762 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:43:25.762 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:43:25.762 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-09-15T20:43:25.762 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2024-09-15T20:43:25.762 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:43:25.762 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:43:25.763 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:43:25.763 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:43:25.763 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:43:25.886 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:43:25.890+0000 7f355aba07c0 -1 Falling back to public interface 2024-09-15T20:43:26.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:43:26.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2024-09-15T20:43:26.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-15T20:43:26.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:43:26.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-15T20:43:26.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:43:26.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2024-09-15T20:43:26.073 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2024-09-15T20:43:27.758 INFO:tasks.workunit.client.0.smithi139.stderr:PG '1.0' not found 2024-09-15T20:43:28.295 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2024-09-15T20:43:28.295 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-09-15T20:43:28.295 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-15T20:43:28.295 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:43:28.295 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T20:43:28.295 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:43:28.295 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-15T20:43:28.295 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:43:28.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:43:28.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:43:28.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:43:28.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-15T20:43:28.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-15T20:43:28.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:43:28.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:43:28.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:43:28.297 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:43:28.297 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:43:28.297 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:43:28.297 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:43:28.297 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:43:28.297 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:43:28.297 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:43:28.299 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:43:28.299 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:43:28.299 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:43:28.299 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:43:28.299 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:43:28.299 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:43:28.299 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:43:28.299 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:43:28.300 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:43:28.300 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:43:28.300 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-15T20:43:28.300 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T20:43:28.300 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.2 2024-09-15T20:43:28.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:43:28.301 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-15T20:43:28.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T20:43:28.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:43:28.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:43:28.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:43:28.351 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:43:28.354+0000 7fdf876e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:43:28.354 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:43:28.358+0000 7fdf876e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:43:28.355 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:43:28.358+0000 7fdf876e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:43:28.495 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:43:28.498+0000 7f355aba07c0 -1 osd.0 43 log_to_monitors true 2024-09-15T20:43:28.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T20:43:28.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:43:28.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:43:28.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:43:28.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:43:28.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:43:28.779 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:43:28.779 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:43:28.780 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:43:28.780 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:43:29.232 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up in weight 1 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/85082059,v1:127.0.0.1:6811/85082059] [v2:127.0.0.1:6812/85082059,v1:127.0.0.1:6813/85082059] exists,up e0a289bd-3c1f-48e6-b612-9a2c75ec33bb 2024-09-15T20:43:29.233 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:43:29.233 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:43:29.233 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:43:29.233 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2024-09-15T20:43:29.233 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-15T20:43:29.234 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-15T20:43:29.234 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-15T20:43:29.234 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-15T20:43:29.234 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-15T20:43:29.234 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2024-09-15T20:43:30.074 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:43:30.077+0000 7fdf876e27c0 -1 Falling back to public interface 2024-09-15T20:43:30.982 INFO:tasks.workunit.client.0.smithi139.stderr:PG '1.0' not found 2024-09-15T20:43:31.542 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2024-09-15T20:43:31.542 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T20:43:31.542 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2024-09-15T20:43:31.542 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:43:31.542 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:43:31.542 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:43:31.543 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T20:43:31.543 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:43:31.543 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:43:31.544 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:43:31.544 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:43:31.544 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:43:31.616 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T20:43:31.616 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:43:31.616 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:43:31.616 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:43:31.616 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:43:31.616 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:43:32.114 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:43:32.114 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:43:32.114 INFO:tasks.workunit.client.0.smithi139.stderr:2' 2024-09-15T20:43:32.114 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:43:32.114 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:43:32.114 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:43:32.297 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=197568495619 2024-09-15T20:43:32.297 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 197568495619 2024-09-15T20:43:32.298 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-197568495619' 2024-09-15T20:43:32.298 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:43:32.298 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:43:32.420 INFO:tasks.workunit.client.0.smithi139.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-15T20:43:32.422 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-15T20:43:32.422 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-15T20:43:32.422 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-15T20:43:32.422 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:43:32.423 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:43:33.406 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:43:33.409+0000 7fdf876e27c0 -1 osd.2 43 log_to_monitors true 2024-09-15T20:43:37.603 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364803 2024-09-15T20:43:37.603 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364803 2024-09-15T20:43:37.603 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-197568495619 2-214748364803' 2024-09-15T20:43:37.603 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:43:37.604 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-197568495619 2024-09-15T20:43:37.604 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:43:37.605 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:43:37.606 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-197568495619 2024-09-15T20:43:37.606 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:43:37.608 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=197568495619 2024-09-15T20:43:37.608 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 197568495619' 2024-09-15T20:43:37.608 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 197568495619 2024-09-15T20:43:37.608 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:43:38.054 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 197568495620 -lt 197568495619 2024-09-15T20:43:38.054 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:43:38.055 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-214748364803 2024-09-15T20:43:38.055 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:43:38.057 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:43:38.058 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-214748364803 2024-09-15T20:43:38.058 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:43:38.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364803 2024-09-15T20:43:38.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 214748364803' 2024-09-15T20:43:38.060 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 214748364803 2024-09-15T20:43:38.060 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:43:38.542 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 214748364803 2024-09-15T20:43:38.542 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:43:39.544 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:43:39.544 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:43:39.991 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364804 -lt 214748364803 2024-09-15T20:43:39.992 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:43:39.993 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:43:39.993 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:43:40.571 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:43:40.571 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:43:40.572 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:43:40.572 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:43:40.572 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:43:40.572 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:43:40.572 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:43:40.573 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:43:41.018 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:43:41.019 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:43:41.019 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:43:41.019 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:43:41.566 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:43:41.567 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-15T20:43:41.567 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-15T20:43:41.567 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-15T20:43:41.567 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:43:41.567 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-15T20:43:41.667 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:43:41.668 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:43:41.668 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:43:41.668 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:43:41.669 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:43:41.669 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:43:41.669 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:43:41.669 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:43:42.117 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:43:42.118 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:43:42.118 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:43:42.118 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:43:42.753 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:43:42.753 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:43:42.753 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:43:42.753 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:43:42.753 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:43:42.753 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:43:42.753 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:43:42.753 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:43:42.754 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:43:43.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:43:43.297 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:43:43.297 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-15T20:43:43.297 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:43:43.297 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-15T20:43:43.498 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:43:43.498 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:43:43.498 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:43:43.498 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:43:43.499 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:43:43.499 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:43:43.499 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:43:43.499 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:43:43.937 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:43:43.937 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:43:43.937 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:43:43.937 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:43:44.510 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:43:44.511 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:43:44.511 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:43:44.511 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:43:44.511 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:43:44.511 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:43:44.511 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:43:44.511 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:43:44.511 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:43:45.111 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:43:45.111 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:43:45.111 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-15T20:43:45.111 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:43:45.111 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-15T20:43:45.512 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:43:45.513 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:43:45.513 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:43:45.514 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:43:45.514 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:43:45.514 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:43:45.514 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:43:45.514 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:43:45.953 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:43:45.953 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:43:45.954 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:43:45.954 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:43:46.518 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:43:46.518 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:43:46.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:43:46.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:43:46.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:43:46.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:43:46.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:43:46.519 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:43:46.519 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:43:47.107 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:43:47.107 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:43:47.107 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-15T20:43:47.108 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:43:47.108 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-15T20:43:47.909 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:43:47.909 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:43:47.910 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:43:47.910 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:43:47.910 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:43:47.911 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:43:47.911 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:43:47.911 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:43:48.331 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:43:48.332 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:43:48.332 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:43:48.333 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:43:48.881 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:43:48.881 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:43:48.881 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:43:48.882 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:43:48.882 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:43:48.882 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:43:48.882 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:43:48.882 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:43:48.882 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:43:49.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:43:49.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:43:49.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-15T20:43:49.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:43:49.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-15T20:43:51.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:43:51.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:43:51.122 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:43:51.122 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:43:51.122 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:43:51.122 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:43:51.122 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:43:51.123 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:43:51.562 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:43:51.563 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:43:51.563 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:43:51.563 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:43:52.136 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:43:52.137 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:43:52.137 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:43:52.137 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:43:52.137 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:43:52.137 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:43:52.137 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:43:52.138 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:43:52.138 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:43:52.697 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:43:52.697 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:43:52.697 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-15T20:43:52.697 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:43:52.697 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-15T20:43:55.898 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:43:55.898 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:43:55.899 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:43:55.899 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:43:55.899 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:43:55.899 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:43:55.900 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:43:55.900 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:43:56.327 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:43:56.327 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:43:56.327 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:43:56.328 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:43:56.873 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:43:56.873 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:43:56.873 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:43:56.874 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:43:56.874 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:43:56.874 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:43:56.874 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:43:56.874 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:43:56.874 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:43:57.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:43:57.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:43:57.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-15T20:43:57.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:43:57.444 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-15T20:44:03.845 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:44:03.845 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:44:03.846 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:44:03.846 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:44:03.846 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:44:03.846 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:44:03.846 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:44:03.847 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:44:04.282 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:44:04.283 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:44:04.283 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:44:04.284 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:44:04.855 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:44:04.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:44:04.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:44:04.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:44:04.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:44:04.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:44:04.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:44:04.857 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:44:04.857 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:44:05.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:44:05.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:44:05.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-15T20:44:05.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-15T20:44:05.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-15T20:44:12.820 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-15T20:44:12.821 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:44:12.821 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:44:12.821 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:44:12.822 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:44:12.822 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:44:12.822 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:44:12.822 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:44:13.262 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-15T20:44:13.263 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:44:13.263 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:44:13.263 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:44:13.837 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-15T20:44:13.837 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-15T20:44:13.837 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-15T20:44:13.837 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-15T20:44:13.837 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T20:44:13.837 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T20:44:13.838 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T20:44:13.838 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-15T20:44:13.838 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T20:44:14.400 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-15T20:44:14.400 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-15T20:44:14.400 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-15T20:44:14.400 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-15T20:44:14.944 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:44:14.944 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_fingerprint": "5ac362c1-a7e8-4310-8ba0-1d9708f202c3", 2024-09-15T20:44:14.944 INFO:tasks.workunit.client.0.smithi139.stdout: "version": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:44:14.944 INFO:tasks.workunit.client.0.smithi139.stdout: "commit": "fc3e0bff7468fd55478adbae7f75508b749124b6", 2024-09-15T20:44:14.944 INFO:tasks.workunit.client.0.smithi139.stdout: "timestamp": "2024-09-15T20:44:14.946913+0000", 2024-09-15T20:44:14.944 INFO:tasks.workunit.client.0.smithi139.stdout: "tag": "", 2024-09-15T20:44:14.945 INFO:tasks.workunit.client.0.smithi139.stdout: "health": { 2024-09-15T20:44:14.945 INFO:tasks.workunit.client.0.smithi139.stdout: "status": "HEALTH_WARN", 2024-09-15T20:44:14.945 INFO:tasks.workunit.client.0.smithi139.stdout: "checks": { 2024-09-15T20:44:14.945 INFO:tasks.workunit.client.0.smithi139.stdout: "OSDMAP_FLAGS": { 2024-09-15T20:44:14.945 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:44:14.945 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:44:14.945 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-15T20:44:14.945 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 17 2024-09-15T20:44:14.945 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.945 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [], 2024-09-15T20:44:14.945 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:44:14.945 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.945 INFO:tasks.workunit.client.0.smithi139.stdout: "OSD_DOWN": { 2024-09-15T20:44:14.946 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:44:14.946 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:44:14.946 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "1 osds down", 2024-09-15T20:44:14.946 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 1 2024-09-15T20:44:14.946 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.946 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:44:14.946 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.946 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "osd.1 (root=default,host=smithi139) is down" 2024-09-15T20:44:14.946 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.946 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:14.946 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:44:14.946 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.947 INFO:tasks.workunit.client.0.smithi139.stdout: "PG_DEGRADED": { 2024-09-15T20:44:14.947 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:44:14.947 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:44:14.947 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-09-15T20:44:14.947 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 1 2024-09-15T20:44:14.947 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.947 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:44:14.947 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.947 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-09-15T20:44:14.947 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.947 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:14.947 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:44:14.947 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.947 INFO:tasks.workunit.client.0.smithi139.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-15T20:44:14.948 INFO:tasks.workunit.client.0.smithi139.stdout: "severity": "HEALTH_WARN", 2024-09-15T20:44:14.948 INFO:tasks.workunit.client.0.smithi139.stdout: "summary": { 2024-09-15T20:44:14.948 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-15T20:44:14.948 INFO:tasks.workunit.client.0.smithi139.stdout: "count": 1 2024-09-15T20:44:14.948 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.948 INFO:tasks.workunit.client.0.smithi139.stdout: "detail": [ 2024-09-15T20:44:14.948 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.948 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "application not enabled on pool 'test'" 2024-09-15T20:44:14.948 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.948 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.948 INFO:tasks.workunit.client.0.smithi139.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-15T20:44:14.948 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.948 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:14.948 INFO:tasks.workunit.client.0.smithi139.stdout: "muted": false 2024-09-15T20:44:14.949 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.949 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.949 INFO:tasks.workunit.client.0.smithi139.stdout: "mutes": [] 2024-09-15T20:44:14.949 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.949 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap_first_committed": 1, 2024-09-15T20:44:14.949 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap_last_committed": 1, 2024-09-15T20:44:14.949 INFO:tasks.workunit.client.0.smithi139.stdout: "monmap": { 2024-09-15T20:44:14.949 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 1, 2024-09-15T20:44:14.949 INFO:tasks.workunit.client.0.smithi139.stdout: "fsid": "0e220df1-ac9c-433e-8885-cf9f7aab293e", 2024-09-15T20:44:14.949 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:39:24.243736Z", 2024-09-15T20:44:14.949 INFO:tasks.workunit.client.0.smithi139.stdout: "created": "2024-09-15T20:39:24.243736Z", 2024-09-15T20:44:14.949 INFO:tasks.workunit.client.0.smithi139.stdout: "min_mon_release": 19, 2024-09-15T20:44:14.949 INFO:tasks.workunit.client.0.smithi139.stdout: "min_mon_release_name": "squid", 2024-09-15T20:44:14.949 INFO:tasks.workunit.client.0.smithi139.stdout: "election_strategy": 1, 2024-09-15T20:44:14.950 INFO:tasks.workunit.client.0.smithi139.stdout: "disallowed_leaders: ": "", 2024-09-15T20:44:14.950 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode": false, 2024-09-15T20:44:14.950 INFO:tasks.workunit.client.0.smithi139.stdout: "tiebreaker_mon": "", 2024-09-15T20:44:14.950 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_ranks: ": "", 2024-09-15T20:44:14.950 INFO:tasks.workunit.client.0.smithi139.stdout: "features": { 2024-09-15T20:44:14.950 INFO:tasks.workunit.client.0.smithi139.stdout: "persistent": [ 2024-09-15T20:44:14.950 INFO:tasks.workunit.client.0.smithi139.stdout: "kraken", 2024-09-15T20:44:14.950 INFO:tasks.workunit.client.0.smithi139.stdout: "luminous", 2024-09-15T20:44:14.950 INFO:tasks.workunit.client.0.smithi139.stdout: "mimic", 2024-09-15T20:44:14.950 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap-prune", 2024-09-15T20:44:14.950 INFO:tasks.workunit.client.0.smithi139.stdout: "nautilus", 2024-09-15T20:44:14.950 INFO:tasks.workunit.client.0.smithi139.stdout: "octopus", 2024-09-15T20:44:14.950 INFO:tasks.workunit.client.0.smithi139.stdout: "pacific", 2024-09-15T20:44:14.950 INFO:tasks.workunit.client.0.smithi139.stdout: "elector-pinging", 2024-09-15T20:44:14.951 INFO:tasks.workunit.client.0.smithi139.stdout: "quincy", 2024-09-15T20:44:14.951 INFO:tasks.workunit.client.0.smithi139.stdout: "reef", 2024-09-15T20:44:14.951 INFO:tasks.workunit.client.0.smithi139.stdout: "squid" 2024-09-15T20:44:14.951 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:14.951 INFO:tasks.workunit.client.0.smithi139.stdout: "optional": [] 2024-09-15T20:44:14.951 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.951 INFO:tasks.workunit.client.0.smithi139.stdout: "mons": [ 2024-09-15T20:44:14.951 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.951 INFO:tasks.workunit.client.0.smithi139.stdout: "rank": 0, 2024-09-15T20:44:14.951 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "a", 2024-09-15T20:44:14.951 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:44:14.951 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:44:14.951 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.952 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:44:14.952 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:7115", 2024-09-15T20:44:14.952 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 0 2024-09-15T20:44:14.952 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.952 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:14.952 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.952 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:7115/0", 2024-09-15T20:44:14.952 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-15T20:44:14.952 INFO:tasks.workunit.client.0.smithi139.stdout: "priority": 0, 2024-09-15T20:44:14.952 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 0, 2024-09-15T20:44:14.952 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_location": "{}" 2024-09-15T20:44:14.952 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.952 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:14.953 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.953 INFO:tasks.workunit.client.0.smithi139.stdout: "quorum": [ 2024-09-15T20:44:14.953 INFO:tasks.workunit.client.0.smithi139.stdout: 0 2024-09-15T20:44:14.953 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:14.953 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap": { 2024-09-15T20:44:14.953 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 52, 2024-09-15T20:44:14.953 INFO:tasks.workunit.client.0.smithi139.stdout: "fsid": "0e220df1-ac9c-433e-8885-cf9f7aab293e", 2024-09-15T20:44:14.953 INFO:tasks.workunit.client.0.smithi139.stdout: "created": "2024-09-15T20:39:25.208770+0000", 2024-09-15T20:44:14.953 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:43:34.696763+0000", 2024-09-15T20:44:14.953 INFO:tasks.workunit.client.0.smithi139.stdout: "last_up_change": "2024-09-15T20:43:33.867772+0000", 2024-09-15T20:44:14.953 INFO:tasks.workunit.client.0.smithi139.stdout: "last_in_change": "2024-09-15T20:42:38.459878+0000", 2024-09-15T20:44:14.953 INFO:tasks.workunit.client.0.smithi139.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T20:44:14.953 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_num": 5800384, 2024-09-15T20:44:14.954 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_set": [ 2024-09-15T20:44:14.954 INFO:tasks.workunit.client.0.smithi139.stdout: "nodown", 2024-09-15T20:44:14.954 INFO:tasks.workunit.client.0.smithi139.stdout: "noin", 2024-09-15T20:44:14.954 INFO:tasks.workunit.client.0.smithi139.stdout: "noout", 2024-09-15T20:44:14.954 INFO:tasks.workunit.client.0.smithi139.stdout: "pglog_hardlimit", 2024-09-15T20:44:14.954 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snapdirs", 2024-09-15T20:44:14.954 INFO:tasks.workunit.client.0.smithi139.stdout: "recovery_deletes", 2024-09-15T20:44:14.954 INFO:tasks.workunit.client.0.smithi139.stdout: "sortbitwise" 2024-09-15T20:44:14.954 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:14.954 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_version": 12, 2024-09-15T20:44:14.954 INFO:tasks.workunit.client.0.smithi139.stdout: "full_ratio": 0.99000000953674316, 2024-09-15T20:44:14.954 INFO:tasks.workunit.client.0.smithi139.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-15T20:44:14.954 INFO:tasks.workunit.client.0.smithi139.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T20:44:14.954 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_snapshot": "", 2024-09-15T20:44:14.955 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_max": 1, 2024-09-15T20:44:14.955 INFO:tasks.workunit.client.0.smithi139.stdout: "max_osd": 3, 2024-09-15T20:44:14.955 INFO:tasks.workunit.client.0.smithi139.stdout: "require_min_compat_client": "luminous", 2024-09-15T20:44:14.955 INFO:tasks.workunit.client.0.smithi139.stdout: "min_compat_client": "jewel", 2024-09-15T20:44:14.955 INFO:tasks.workunit.client.0.smithi139.stdout: "require_osd_release": "squid", 2024-09-15T20:44:14.955 INFO:tasks.workunit.client.0.smithi139.stdout: "allow_crimson": false, 2024-09-15T20:44:14.955 INFO:tasks.workunit.client.0.smithi139.stdout: "pools": [ 2024-09-15T20:44:14.955 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.955 INFO:tasks.workunit.client.0.smithi139.stdout: "pool": 1, 2024-09-15T20:44:14.955 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_name": "test", 2024-09-15T20:44:14.955 INFO:tasks.workunit.client.0.smithi139.stdout: "create_time": "2024-09-15T20:41:23.859990+0000", 2024-09-15T20:44:14.955 INFO:tasks.workunit.client.0.smithi139.stdout: "flags": 16389, 2024-09-15T20:44:14.955 INFO:tasks.workunit.client.0.smithi139.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-09-15T20:44:14.955 INFO:tasks.workunit.client.0.smithi139.stdout: "type": 3, 2024-09-15T20:44:14.956 INFO:tasks.workunit.client.0.smithi139.stdout: "size": 3, 2024-09-15T20:44:14.956 INFO:tasks.workunit.client.0.smithi139.stdout: "min_size": 2, 2024-09-15T20:44:14.956 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_rule": 1, 2024-09-15T20:44:14.956 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_count": 0, 2024-09-15T20:44:14.956 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_target": 0, 2024-09-15T20:44:14.956 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T20:44:14.956 INFO:tasks.workunit.client.0.smithi139.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T20:44:14.956 INFO:tasks.workunit.client.0.smithi139.stdout: "is_stretch_pool": false, 2024-09-15T20:44:14.956 INFO:tasks.workunit.client.0.smithi139.stdout: "object_hash": 2, 2024-09-15T20:44:14.956 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_autoscale_mode": "off", 2024-09-15T20:44:14.956 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num": 1, 2024-09-15T20:44:14.956 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_placement_num": 1, 2024-09-15T20:44:14.956 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_placement_num_target": 1, 2024-09-15T20:44:14.957 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num_target": 1, 2024-09-15T20:44:14.957 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_num_pending": 1, 2024-09-15T20:44:14.957 INFO:tasks.workunit.client.0.smithi139.stdout: "last_pg_merge_meta": { 2024-09-15T20:44:14.957 INFO:tasks.workunit.client.0.smithi139.stdout: "source_pgid": "0.0", 2024-09-15T20:44:14.957 INFO:tasks.workunit.client.0.smithi139.stdout: "ready_epoch": 0, 2024-09-15T20:44:14.957 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_started": 0, 2024-09-15T20:44:14.957 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_clean": 0, 2024-09-15T20:44:14.957 INFO:tasks.workunit.client.0.smithi139.stdout: "source_version": "0'0", 2024-09-15T20:44:14.957 INFO:tasks.workunit.client.0.smithi139.stdout: "target_version": "0'0" 2024-09-15T20:44:14.957 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.957 INFO:tasks.workunit.client.0.smithi139.stdout: "last_change": "30", 2024-09-15T20:44:14.957 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend": "0", 2024-09-15T20:44:14.957 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T20:44:14.957 INFO:tasks.workunit.client.0.smithi139.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T20:44:14.958 INFO:tasks.workunit.client.0.smithi139.stdout: "auid": 0, 2024-09-15T20:44:14.958 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_mode": "pool", 2024-09-15T20:44:14.958 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_seq": 1, 2024-09-15T20:44:14.958 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_epoch": 30, 2024-09-15T20:44:14.958 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_snaps": [ 2024-09-15T20:44:14.958 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.958 INFO:tasks.workunit.client.0.smithi139.stdout: "snapid": 1, 2024-09-15T20:44:14.958 INFO:tasks.workunit.client.0.smithi139.stdout: "stamp": "2024-09-15T20:42:12.247870+0000", 2024-09-15T20:44:14.958 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "snap1" 2024-09-15T20:44:14.958 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.958 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:14.958 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_snaps": "[]", 2024-09-15T20:44:14.958 INFO:tasks.workunit.client.0.smithi139.stdout: "quota_max_bytes": 0, 2024-09-15T20:44:14.958 INFO:tasks.workunit.client.0.smithi139.stdout: "quota_max_objects": 0, 2024-09-15T20:44:14.959 INFO:tasks.workunit.client.0.smithi139.stdout: "tiers": [], 2024-09-15T20:44:14.959 INFO:tasks.workunit.client.0.smithi139.stdout: "tier_of": -1, 2024-09-15T20:44:14.959 INFO:tasks.workunit.client.0.smithi139.stdout: "read_tier": -1, 2024-09-15T20:44:14.959 INFO:tasks.workunit.client.0.smithi139.stdout: "write_tier": -1, 2024-09-15T20:44:14.959 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_mode": "none", 2024-09-15T20:44:14.959 INFO:tasks.workunit.client.0.smithi139.stdout: "target_max_bytes": 0, 2024-09-15T20:44:14.959 INFO:tasks.workunit.client.0.smithi139.stdout: "target_max_objects": 0, 2024-09-15T20:44:14.959 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T20:44:14.959 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T20:44:14.959 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T20:44:14.959 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_min_flush_age": 0, 2024-09-15T20:44:14.959 INFO:tasks.workunit.client.0.smithi139.stdout: "cache_min_evict_age": 0, 2024-09-15T20:44:14.959 INFO:tasks.workunit.client.0.smithi139.stdout: "erasure_code_profile": "myprofile", 2024-09-15T20:44:14.959 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_params": { 2024-09-15T20:44:14.959 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "none" 2024-09-15T20:44:14.960 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.960 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_period": 0, 2024-09-15T20:44:14.960 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_count": 0, 2024-09-15T20:44:14.960 INFO:tasks.workunit.client.0.smithi139.stdout: "use_gmt_hitset": true, 2024-09-15T20:44:14.960 INFO:tasks.workunit.client.0.smithi139.stdout: "min_read_recency_for_promote": 0, 2024-09-15T20:44:14.960 INFO:tasks.workunit.client.0.smithi139.stdout: "min_write_recency_for_promote": 0, 2024-09-15T20:44:14.960 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T20:44:14.960 INFO:tasks.workunit.client.0.smithi139.stdout: "hit_set_search_last_n": 0, 2024-09-15T20:44:14.960 INFO:tasks.workunit.client.0.smithi139.stdout: "grade_table": [], 2024-09-15T20:44:14.960 INFO:tasks.workunit.client.0.smithi139.stdout: "stripe_width": 8192, 2024-09-15T20:44:14.960 INFO:tasks.workunit.client.0.smithi139.stdout: "expected_num_objects": 0, 2024-09-15T20:44:14.960 INFO:tasks.workunit.client.0.smithi139.stdout: "fast_read": false, 2024-09-15T20:44:14.960 INFO:tasks.workunit.client.0.smithi139.stdout: "options": {}, 2024-09-15T20:44:14.960 INFO:tasks.workunit.client.0.smithi139.stdout: "application_metadata": {} 2024-09-15T20:44:14.960 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.961 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:14.961 INFO:tasks.workunit.client.0.smithi139.stdout: "osds": [ 2024-09-15T20:44:14.961 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.961 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:44:14.961 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "072ef209-0424-4dcb-ba88-59a19eaa46bd", 2024-09-15T20:44:14.961 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 1, 2024-09-15T20:44:14.961 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:44:14.961 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:44:14.961 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:44:14.961 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 37, 2024-09-15T20:44:14.961 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 43, 2024-09-15T20:44:14.961 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 46, 2024-09-15T20:44:14.961 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 51, 2024-09-15T20:44:14.961 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 45, 2024-09-15T20:44:14.962 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:44:14.962 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:44:14.962 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:44:14.962 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.962 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:44:14.962 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6802", 2024-09-15T20:44:14.962 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 463017228 2024-09-15T20:44:14.962 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.962 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.962 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:44:14.962 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6803", 2024-09-15T20:44:14.962 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 463017228 2024-09-15T20:44:14.962 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.962 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:14.962 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.963 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:44:14.963 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:44:14.963 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.963 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:44:14.963 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6804", 2024-09-15T20:44:14.963 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 463017228 2024-09-15T20:44:14.963 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.963 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.963 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:44:14.963 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6805", 2024-09-15T20:44:14.963 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 463017228 2024-09-15T20:44:14.963 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.963 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:14.963 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.964 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:44:14.964 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:44:14.964 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.964 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:44:14.964 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6808", 2024-09-15T20:44:14.964 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 463017228 2024-09-15T20:44:14.964 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.964 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.964 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:44:14.964 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6809", 2024-09-15T20:44:14.964 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 463017228 2024-09-15T20:44:14.964 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.964 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:14.964 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.965 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:44:14.965 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:44:14.965 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.965 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:44:14.965 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6806", 2024-09-15T20:44:14.965 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 463017228 2024-09-15T20:44:14.965 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.965 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.965 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:44:14.965 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6807", 2024-09-15T20:44:14.966 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 463017228 2024-09-15T20:44:14.966 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.966 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:14.966 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.966 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6803/463017228", 2024-09-15T20:44:14.966 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6805/463017228", 2024-09-15T20:44:14.966 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6809/463017228", 2024-09-15T20:44:14.966 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6807/463017228", 2024-09-15T20:44:14.966 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:44:14.966 INFO:tasks.workunit.client.0.smithi139.stdout: "exists", 2024-09-15T20:44:14.966 INFO:tasks.workunit.client.0.smithi139.stdout: "up" 2024-09-15T20:44:14.967 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:14.967 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.967 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.967 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:44:14.967 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "61477bed-b55a-47e6-8ac3-ac4772590dce", 2024-09-15T20:44:14.967 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 0, 2024-09-15T20:44:14.967 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:44:14.967 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:44:14.967 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:44:14.967 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 0, 2024-09-15T20:44:14.967 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 0, 2024-09-15T20:44:14.967 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 10, 2024-09-15T20:44:14.967 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 25, 2024-09-15T20:44:14.967 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 32, 2024-09-15T20:44:14.968 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:44:14.968 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:44:14.968 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:44:14.968 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.968 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:44:14.968 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6810", 2024-09-15T20:44:14.968 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3833330082 2024-09-15T20:44:14.968 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.968 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.968 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:44:14.968 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6811", 2024-09-15T20:44:14.968 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3833330082 2024-09-15T20:44:14.968 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.968 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:14.968 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.969 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:44:14.969 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:44:14.969 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.969 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:44:14.969 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6812", 2024-09-15T20:44:14.969 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3833330082 2024-09-15T20:44:14.969 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.969 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.969 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:44:14.969 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6813", 2024-09-15T20:44:14.969 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3833330082 2024-09-15T20:44:14.969 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.969 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:14.969 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.970 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:44:14.970 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:44:14.970 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.970 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:44:14.970 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6816", 2024-09-15T20:44:14.970 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3833330082 2024-09-15T20:44:14.970 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.970 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.970 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:44:14.970 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6817", 2024-09-15T20:44:14.970 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3833330082 2024-09-15T20:44:14.970 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.970 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:14.971 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.971 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:44:14.971 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:44:14.971 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.971 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:44:14.971 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6814", 2024-09-15T20:44:14.971 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3833330082 2024-09-15T20:44:14.971 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.971 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.971 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:44:14.971 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6815", 2024-09-15T20:44:14.971 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 3833330082 2024-09-15T20:44:14.971 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.971 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:14.972 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.972 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6811/3833330082", 2024-09-15T20:44:14.972 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6813/3833330082", 2024-09-15T20:44:14.972 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3833330082", 2024-09-15T20:44:14.972 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3833330082", 2024-09-15T20:44:14.972 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:44:14.972 INFO:tasks.workunit.client.0.smithi139.stdout: "exists" 2024-09-15T20:44:14.972 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:14.972 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.972 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.972 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:44:14.972 INFO:tasks.workunit.client.0.smithi139.stdout: "uuid": "e0a289bd-3c1f-48e6-b612-9a2c75ec33bb", 2024-09-15T20:44:14.972 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 1, 2024-09-15T20:44:14.972 INFO:tasks.workunit.client.0.smithi139.stdout: "in": 1, 2024-09-15T20:44:14.973 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 1, 2024-09-15T20:44:14.973 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_affinity": 1, 2024-09-15T20:44:14.973 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_begin": 39, 2024-09-15T20:44:14.973 INFO:tasks.workunit.client.0.smithi139.stdout: "last_clean_end": 43, 2024-09-15T20:44:14.973 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 50, 2024-09-15T20:44:14.973 INFO:tasks.workunit.client.0.smithi139.stdout: "up_thru": 0, 2024-09-15T20:44:14.973 INFO:tasks.workunit.client.0.smithi139.stdout: "down_at": 49, 2024-09-15T20:44:14.973 INFO:tasks.workunit.client.0.smithi139.stdout: "lost_at": 0, 2024-09-15T20:44:14.973 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addrs": { 2024-09-15T20:44:14.973 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:44:14.973 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.973 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:44:14.973 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6810", 2024-09-15T20:44:14.973 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 430831352 2024-09-15T20:44:14.974 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.974 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.974 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:44:14.974 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6811", 2024-09-15T20:44:14.974 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 430831352 2024-09-15T20:44:14.974 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.974 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:14.974 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.974 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addrs": { 2024-09-15T20:44:14.975 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:44:14.975 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.975 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:44:14.975 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6812", 2024-09-15T20:44:14.975 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 430831352 2024-09-15T20:44:14.975 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.975 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.975 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:44:14.975 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6813", 2024-09-15T20:44:14.975 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 430831352 2024-09-15T20:44:14.975 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.975 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:14.975 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.975 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addrs": { 2024-09-15T20:44:14.976 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:44:14.976 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.976 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:44:14.976 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6816", 2024-09-15T20:44:14.976 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 430831352 2024-09-15T20:44:14.976 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.976 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.976 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:44:14.976 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6817", 2024-09-15T20:44:14.976 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 430831352 2024-09-15T20:44:14.976 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.976 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:14.976 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.976 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addrs": { 2024-09-15T20:44:14.977 INFO:tasks.workunit.client.0.smithi139.stdout: "addrvec": [ 2024-09-15T20:44:14.977 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.977 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v2", 2024-09-15T20:44:14.977 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6814", 2024-09-15T20:44:14.977 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 430831352 2024-09-15T20:44:14.977 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.977 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.977 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "v1", 2024-09-15T20:44:14.977 INFO:tasks.workunit.client.0.smithi139.stdout: "addr": "127.0.0.1:6815", 2024-09-15T20:44:14.977 INFO:tasks.workunit.client.0.smithi139.stdout: "nonce": 430831352 2024-09-15T20:44:14.977 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.977 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:14.977 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.977 INFO:tasks.workunit.client.0.smithi139.stdout: "public_addr": "127.0.0.1:6811/430831352", 2024-09-15T20:44:14.977 INFO:tasks.workunit.client.0.smithi139.stdout: "cluster_addr": "127.0.0.1:6813/430831352", 2024-09-15T20:44:14.978 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_back_addr": "127.0.0.1:6817/430831352", 2024-09-15T20:44:14.978 INFO:tasks.workunit.client.0.smithi139.stdout: "heartbeat_front_addr": "127.0.0.1:6815/430831352", 2024-09-15T20:44:14.978 INFO:tasks.workunit.client.0.smithi139.stdout: "state": [ 2024-09-15T20:44:14.978 INFO:tasks.workunit.client.0.smithi139.stdout: "exists", 2024-09-15T20:44:14.978 INFO:tasks.workunit.client.0.smithi139.stdout: "up" 2024-09-15T20:44:14.978 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:14.978 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.978 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:14.978 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_xinfo": [ 2024-09-15T20:44:14.978 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.978 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:44:14.978 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:43:28.803993+0000", 2024-09-15T20:44:14.978 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:44:14.978 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:44:14.979 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:44:14.979 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:44:14.979 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:40:01.324476+0000", 2024-09-15T20:44:14.979 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:44:14.979 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.979 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.979 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:44:14.979 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:42:24.031556+0000", 2024-09-15T20:44:14.979 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:44:14.979 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:44:14.979 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:44:14.979 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:44:14.979 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:40:39.910012+0000", 2024-09-15T20:44:14.980 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:44:14.980 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.980 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.980 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:44:14.980 INFO:tasks.workunit.client.0.smithi139.stdout: "down_stamp": "2024-09-15T20:43:33.658609+0000", 2024-09-15T20:44:14.980 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_probability": 0, 2024-09-15T20:44:14.980 INFO:tasks.workunit.client.0.smithi139.stdout: "laggy_interval": 0, 2024-09-15T20:44:14.980 INFO:tasks.workunit.client.0.smithi139.stdout: "features": 4540719139924082687, 2024-09-15T20:44:14.980 INFO:tasks.workunit.client.0.smithi139.stdout: "old_weight": 0, 2024-09-15T20:44:14.980 INFO:tasks.workunit.client.0.smithi139.stdout: "last_purged_snaps_scrub": "2024-09-15T20:41:18.728508+0000", 2024-09-15T20:44:14.980 INFO:tasks.workunit.client.0.smithi139.stdout: "dead_epoch": 0 2024-09-15T20:44:14.980 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.980 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:14.980 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap": [], 2024-09-15T20:44:14.981 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap_items": [], 2024-09-15T20:44:14.981 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_upmap_primaries": [], 2024-09-15T20:44:14.981 INFO:tasks.workunit.client.0.smithi139.stdout: "pg_temp": [], 2024-09-15T20:44:14.981 INFO:tasks.workunit.client.0.smithi139.stdout: "primary_temp": [], 2024-09-15T20:44:14.981 INFO:tasks.workunit.client.0.smithi139.stdout: "blocklist": {}, 2024-09-15T20:44:14.981 INFO:tasks.workunit.client.0.smithi139.stdout: "range_blocklist": {}, 2024-09-15T20:44:14.981 INFO:tasks.workunit.client.0.smithi139.stdout: "erasure_code_profiles": { 2024-09-15T20:44:14.981 INFO:tasks.workunit.client.0.smithi139.stdout: "default": { 2024-09-15T20:44:14.981 INFO:tasks.workunit.client.0.smithi139.stdout: "k": "2", 2024-09-15T20:44:14.981 INFO:tasks.workunit.client.0.smithi139.stdout: "m": "2", 2024-09-15T20:44:14.981 INFO:tasks.workunit.client.0.smithi139.stdout: "plugin": "jerasure", 2024-09-15T20:44:14.981 INFO:tasks.workunit.client.0.smithi139.stdout: "technique": "reed_sol_van" 2024-09-15T20:44:14.981 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.981 INFO:tasks.workunit.client.0.smithi139.stdout: "myprofile": { 2024-09-15T20:44:14.981 INFO:tasks.workunit.client.0.smithi139.stdout: "crush-device-class": "", 2024-09-15T20:44:14.982 INFO:tasks.workunit.client.0.smithi139.stdout: "crush-failure-domain": "osd", 2024-09-15T20:44:14.982 INFO:tasks.workunit.client.0.smithi139.stdout: "crush-num-failure-domains": "0", 2024-09-15T20:44:14.982 INFO:tasks.workunit.client.0.smithi139.stdout: "crush-osds-per-failure-domain": "0", 2024-09-15T20:44:14.982 INFO:tasks.workunit.client.0.smithi139.stdout: "crush-root": "default", 2024-09-15T20:44:14.982 INFO:tasks.workunit.client.0.smithi139.stdout: "jerasure-per-chunk-alignment": "false", 2024-09-15T20:44:14.982 INFO:tasks.workunit.client.0.smithi139.stdout: "k": "2", 2024-09-15T20:44:14.982 INFO:tasks.workunit.client.0.smithi139.stdout: "m": "1", 2024-09-15T20:44:14.982 INFO:tasks.workunit.client.0.smithi139.stdout: "plugin": "jerasure", 2024-09-15T20:44:14.982 INFO:tasks.workunit.client.0.smithi139.stdout: "technique": "reed_sol_van", 2024-09-15T20:44:14.982 INFO:tasks.workunit.client.0.smithi139.stdout: "w": "8" 2024-09-15T20:44:14.982 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.982 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.982 INFO:tasks.workunit.client.0.smithi139.stdout: "removed_snaps_queue": [], 2024-09-15T20:44:14.982 INFO:tasks.workunit.client.0.smithi139.stdout: "new_removed_snaps": [], 2024-09-15T20:44:14.983 INFO:tasks.workunit.client.0.smithi139.stdout: "new_purged_snaps": [], 2024-09-15T20:44:14.983 INFO:tasks.workunit.client.0.smithi139.stdout: "crush_node_flags": {}, 2024-09-15T20:44:14.983 INFO:tasks.workunit.client.0.smithi139.stdout: "device_class_flags": {}, 2024-09-15T20:44:14.983 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode": { 2024-09-15T20:44:14.983 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode_enabled": false, 2024-09-15T20:44:14.983 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_bucket_count": 0, 2024-09-15T20:44:14.983 INFO:tasks.workunit.client.0.smithi139.stdout: "degraded_stretch_mode": 0, 2024-09-15T20:44:14.983 INFO:tasks.workunit.client.0.smithi139.stdout: "recovering_stretch_mode": 0, 2024-09-15T20:44:14.983 INFO:tasks.workunit.client.0.smithi139.stdout: "stretch_mode_bucket": 0 2024-09-15T20:44:14.983 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.983 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.983 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_metadata": [ 2024-09-15T20:44:14.983 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.983 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:44:14.984 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:44:14.984 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6804/463017228,v1:127.0.0.1:6805/463017228]", 2024-09-15T20:44:14.984 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:44:14.984 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:44:14.984 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:44:14.984 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:44:14.984 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:44:14.984 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:44:14.984 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:44:14.984 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:44:14.984 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:44:14.984 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:44:14.984 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:44:14.985 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-15T20:44:14.985 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:44:14.985 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:44:14.985 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:44:14.985 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:44:14.985 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:44:14.985 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:44:14.985 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:44:14.985 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:44:14.985 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:44:14.985 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:44:14.985 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:39:32.056892Z", 2024-09-15T20:44:14.985 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:44:14.985 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:44:14.986 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:44:14.986 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:44:14.986 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:44:14.986 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:44:14.986 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:44:14.986 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6802/463017228,v1:127.0.0.1:6803/463017228]", 2024-09-15T20:44:14.986 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:44:14.986 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/463017228,v1:127.0.0.1:6809/463017228]", 2024-09-15T20:44:14.986 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/463017228,v1:127.0.0.1:6807/463017228]", 2024-09-15T20:44:14.986 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:44:14.986 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:44:14.986 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:44:14.986 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:44:14.986 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:44:14.987 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:44:14.987 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:44:14.987 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:44:14.987 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:44:14.987 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/0", 2024-09-15T20:44:14.987 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:44:14.987 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:44:14.987 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:44:14.987 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.987 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.987 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:44:14.987 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:44:14.987 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6812/3833330082,v1:127.0.0.1:6813/3833330082]", 2024-09-15T20:44:14.987 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:44:14.988 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:44:14.988 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:44:14.988 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:44:14.988 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:44:14.988 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:44:14.988 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:44:14.988 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:44:14.988 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:44:14.988 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:44:14.988 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:44:14.988 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-15T20:44:14.988 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:44:14.989 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:44:14.989 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:44:14.989 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:44:14.989 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:44:14.989 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:44:14.989 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:44:14.989 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:44:14.989 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:44:14.989 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:44:14.989 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:40:08.814496Z", 2024-09-15T20:44:14.989 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:44:14.990 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:44:14.990 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:44:14.990 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:44:14.990 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:44:14.990 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:44:14.990 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:44:14.990 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6810/3833330082,v1:127.0.0.1:6811/3833330082]", 2024-09-15T20:44:14.990 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:44:14.990 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3833330082,v1:127.0.0.1:6817/3833330082]", 2024-09-15T20:44:14.990 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3833330082,v1:127.0.0.1:6815/3833330082]", 2024-09-15T20:44:14.990 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:44:14.991 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:44:14.991 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:44:14.991 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:44:14.991 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:44:14.991 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:44:14.991 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:44:14.991 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:44:14.991 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:44:14.991 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/1", 2024-09-15T20:44:14.991 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:44:14.991 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:44:14.991 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:44:14.991 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.992 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.992 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:44:14.992 INFO:tasks.workunit.client.0.smithi139.stdout: "arch": "x86_64", 2024-09-15T20:44:14.992 INFO:tasks.workunit.client.0.smithi139.stdout: "back_addr": "[v2:127.0.0.1:6812/430831352,v1:127.0.0.1:6813/430831352]", 2024-09-15T20:44:14.992 INFO:tasks.workunit.client.0.smithi139.stdout: "back_iface": "", 2024-09-15T20:44:14.992 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs": "1", 2024-09-15T20:44:14.992 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_db": "0", 2024-09-15T20:44:14.992 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T20:44:14.992 INFO:tasks.workunit.client.0.smithi139.stdout: "bluefs_single_shared_device": "1", 2024-09-15T20:44:14.992 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T20:44:14.992 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T20:44:14.992 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T20:44:14.992 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T20:44:14.993 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T20:44:14.993 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T20:44:14.993 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-15T20:44:14.993 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T20:44:14.993 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T20:44:14.993 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_support_discard": "0", 2024-09-15T20:44:14.993 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T20:44:14.993 INFO:tasks.workunit.client.0.smithi139.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T20:44:14.993 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_release": "squid", 2024-09-15T20:44:14.993 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:44:14.993 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:44:14.993 INFO:tasks.workunit.client.0.smithi139.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:44:14.994 INFO:tasks.workunit.client.0.smithi139.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:44:14.994 INFO:tasks.workunit.client.0.smithi139.stdout: "created_at": "2024-09-15T20:40:48.254151Z", 2024-09-15T20:44:14.994 INFO:tasks.workunit.client.0.smithi139.stdout: "default_device_class": "hdd", 2024-09-15T20:44:14.994 INFO:tasks.workunit.client.0.smithi139.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XFR4", 2024-09-15T20:44:14.994 INFO:tasks.workunit.client.0.smithi139.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T20:44:14.994 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": "sda", 2024-09-15T20:44:14.994 INFO:tasks.workunit.client.0.smithi139.stdout: "distro": "ubuntu", 2024-09-15T20:44:14.994 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:44:14.994 INFO:tasks.workunit.client.0.smithi139.stdout: "distro_version": "22.04", 2024-09-15T20:44:14.994 INFO:tasks.workunit.client.0.smithi139.stdout: "front_addr": "[v2:127.0.0.1:6810/430831352,v1:127.0.0.1:6811/430831352]", 2024-09-15T20:44:14.994 INFO:tasks.workunit.client.0.smithi139.stdout: "front_iface": "", 2024-09-15T20:44:14.994 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/430831352,v1:127.0.0.1:6817/430831352]", 2024-09-15T20:44:14.994 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/430831352,v1:127.0.0.1:6815/430831352]", 2024-09-15T20:44:14.995 INFO:tasks.workunit.client.0.smithi139.stdout: "hostname": "smithi139", 2024-09-15T20:44:14.995 INFO:tasks.workunit.client.0.smithi139.stdout: "journal_rotational": "1", 2024-09-15T20:44:14.995 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:44:14.995 INFO:tasks.workunit.client.0.smithi139.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:44:14.995 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_swap_kb": "2097148", 2024-09-15T20:44:14.995 INFO:tasks.workunit.client.0.smithi139.stdout: "mem_total_kb": "32746744", 2024-09-15T20:44:14.995 INFO:tasks.workunit.client.0.smithi139.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T20:44:14.995 INFO:tasks.workunit.client.0.smithi139.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T20:44:14.995 INFO:tasks.workunit.client.0.smithi139.stdout: "os": "Linux", 2024-09-15T20:44:14.996 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_data": "td/divergent-priors/2", 2024-09-15T20:44:14.996 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_objectstore": "bluestore", 2024-09-15T20:44:14.996 INFO:tasks.workunit.client.0.smithi139.stdout: "osdspec_affinity": "", 2024-09-15T20:44:14.996 INFO:tasks.workunit.client.0.smithi139.stdout: "rotational": "1" 2024-09-15T20:44:14.996 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.996 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:14.996 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_clean_epochs": { 2024-09-15T20:44:14.996 INFO:tasks.workunit.client.0.smithi139.stdout: "min_last_epoch_clean": 26, 2024-09-15T20:44:14.996 INFO:tasks.workunit.client.0.smithi139.stdout: "last_epoch_clean": { 2024-09-15T20:44:14.996 INFO:tasks.workunit.client.0.smithi139.stdout: "per_pool": [ 2024-09-15T20:44:14.996 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.996 INFO:tasks.workunit.client.0.smithi139.stdout: "poolid": 1, 2024-09-15T20:44:14.996 INFO:tasks.workunit.client.0.smithi139.stdout: "floor": 26 2024-09-15T20:44:14.997 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.997 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:14.997 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.997 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_epochs": [ 2024-09-15T20:44:14.997 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.997 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:44:14.997 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 47 2024-09-15T20:44:14.997 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.997 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.997 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:44:14.997 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 50 2024-09-15T20:44:14.997 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.997 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:14.997 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.998 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_first_committed": 1, 2024-09-15T20:44:14.998 INFO:tasks.workunit.client.0.smithi139.stdout: "osdmap_last_committed": 52, 2024-09-15T20:44:14.998 INFO:tasks.workunit.client.0.smithi139.stdout: "crushmap": { 2024-09-15T20:44:14.998 INFO:tasks.workunit.client.0.smithi139.stdout: "devices": [ 2024-09-15T20:44:14.998 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.998 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:44:14.998 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.0", 2024-09-15T20:44:14.998 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:44:14.998 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.998 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.998 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:44:14.998 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.1", 2024-09-15T20:44:14.998 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:44:14.998 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.999 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.999 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:44:14.999 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd.2", 2024-09-15T20:44:14.999 INFO:tasks.workunit.client.0.smithi139.stdout: "class": "hdd" 2024-09-15T20:44:14.999 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:14.999 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:14.999 INFO:tasks.workunit.client.0.smithi139.stdout: "types": [ 2024-09-15T20:44:14.999 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.999 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 0, 2024-09-15T20:44:14.999 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "osd" 2024-09-15T20:44:14.999 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:14.999 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:14.999 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:44:14.999 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "host" 2024-09-15T20:44:15.000 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.000 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.000 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 2, 2024-09-15T20:44:15.000 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "chassis" 2024-09-15T20:44:15.000 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.000 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.000 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 3, 2024-09-15T20:44:15.000 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "rack" 2024-09-15T20:44:15.000 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.000 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.000 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 4, 2024-09-15T20:44:15.000 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "row" 2024-09-15T20:44:15.000 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.000 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.001 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 5, 2024-09-15T20:44:15.001 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "pdu" 2024-09-15T20:44:15.001 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.001 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.001 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 6, 2024-09-15T20:44:15.001 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "pod" 2024-09-15T20:44:15.001 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.001 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.001 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 7, 2024-09-15T20:44:15.001 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "room" 2024-09-15T20:44:15.001 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.001 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.001 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 8, 2024-09-15T20:44:15.002 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "datacenter" 2024-09-15T20:44:15.002 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.002 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.002 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 9, 2024-09-15T20:44:15.002 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "zone" 2024-09-15T20:44:15.002 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.002 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.002 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 10, 2024-09-15T20:44:15.002 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "region" 2024-09-15T20:44:15.002 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.002 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.002 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:44:15.002 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "root" 2024-09-15T20:44:15.003 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:15.003 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:15.003 INFO:tasks.workunit.client.0.smithi139.stdout: "buckets": [ 2024-09-15T20:44:15.003 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.003 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -1, 2024-09-15T20:44:15.003 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "default", 2024-09-15T20:44:15.003 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:44:15.003 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "root", 2024-09-15T20:44:15.003 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:44:15.003 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:44:15.003 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:44:15.003 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:44:15.003 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.004 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -3, 2024-09-15T20:44:15.004 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:44:15.004 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:44:15.004 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:15.004 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:15.004 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.004 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.004 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -2, 2024-09-15T20:44:15.004 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "default~hdd", 2024-09-15T20:44:15.004 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 11, 2024-09-15T20:44:15.004 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "root", 2024-09-15T20:44:15.004 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:44:15.004 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:44:15.005 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:44:15.005 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:44:15.005 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.005 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -4, 2024-09-15T20:44:15.005 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:44:15.005 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:44:15.005 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:15.005 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:15.006 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.006 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.006 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -3, 2024-09-15T20:44:15.006 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "smithi139", 2024-09-15T20:44:15.006 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:44:15.006 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "host", 2024-09-15T20:44:15.006 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:44:15.006 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:44:15.006 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:44:15.006 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:44:15.006 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.006 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:44:15.006 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:44:15.006 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:44:15.006 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.007 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.007 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:44:15.007 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:44:15.007 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 1 2024-09-15T20:44:15.007 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.007 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.007 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:44:15.007 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:44:15.007 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 2 2024-09-15T20:44:15.007 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:15.007 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:15.007 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.007 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.008 INFO:tasks.workunit.client.0.smithi139.stdout: "id": -4, 2024-09-15T20:44:15.008 INFO:tasks.workunit.client.0.smithi139.stdout: "name": "smithi139~hdd", 2024-09-15T20:44:15.008 INFO:tasks.workunit.client.0.smithi139.stdout: "type_id": 1, 2024-09-15T20:44:15.008 INFO:tasks.workunit.client.0.smithi139.stdout: "type_name": "host", 2024-09-15T20:44:15.008 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 19206, 2024-09-15T20:44:15.008 INFO:tasks.workunit.client.0.smithi139.stdout: "alg": "straw2", 2024-09-15T20:44:15.008 INFO:tasks.workunit.client.0.smithi139.stdout: "hash": "rjenkins1", 2024-09-15T20:44:15.008 INFO:tasks.workunit.client.0.smithi139.stdout: "items": [ 2024-09-15T20:44:15.008 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.008 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 0, 2024-09-15T20:44:15.009 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:44:15.009 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 0 2024-09-15T20:44:15.009 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.009 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.009 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 1, 2024-09-15T20:44:15.009 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:44:15.009 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 1 2024-09-15T20:44:15.009 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.009 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.009 INFO:tasks.workunit.client.0.smithi139.stdout: "id": 2, 2024-09-15T20:44:15.009 INFO:tasks.workunit.client.0.smithi139.stdout: "weight": 6402, 2024-09-15T20:44:15.009 INFO:tasks.workunit.client.0.smithi139.stdout: "pos": 2 2024-09-15T20:44:15.009 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:15.009 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:15.009 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:15.010 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:15.010 INFO:tasks.workunit.client.0.smithi139.stdout: "rules": [ 2024-09-15T20:44:15.010 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.010 INFO:tasks.workunit.client.0.smithi139.stdout: "rule_id": 0, 2024-09-15T20:44:15.010 INFO:tasks.workunit.client.0.smithi139.stdout: "rule_name": "replicated_rule", 2024-09-15T20:44:15.010 INFO:tasks.workunit.client.0.smithi139.stdout: "type": 1, 2024-09-15T20:44:15.010 INFO:tasks.workunit.client.0.smithi139.stdout: "steps": [ 2024-09-15T20:44:15.010 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.010 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "take", 2024-09-15T20:44:15.010 INFO:tasks.workunit.client.0.smithi139.stdout: "item": -1, 2024-09-15T20:44:15.010 INFO:tasks.workunit.client.0.smithi139.stdout: "item_name": "default" 2024-09-15T20:44:15.010 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.010 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.011 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "choose_firstn", 2024-09-15T20:44:15.011 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 0, 2024-09-15T20:44:15.011 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "osd" 2024-09-15T20:44:15.011 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.011 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.011 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "emit" 2024-09-15T20:44:15.011 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:15.011 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:15.011 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.011 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.011 INFO:tasks.workunit.client.0.smithi139.stdout: "rule_id": 1, 2024-09-15T20:44:15.011 INFO:tasks.workunit.client.0.smithi139.stdout: "rule_name": "test", 2024-09-15T20:44:15.011 INFO:tasks.workunit.client.0.smithi139.stdout: "type": 3, 2024-09-15T20:44:15.011 INFO:tasks.workunit.client.0.smithi139.stdout: "steps": [ 2024-09-15T20:44:15.012 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.012 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "set_chooseleaf_tries", 2024-09-15T20:44:15.012 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 5 2024-09-15T20:44:15.012 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.012 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.012 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "set_choose_tries", 2024-09-15T20:44:15.012 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 100 2024-09-15T20:44:15.012 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.012 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.012 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "take", 2024-09-15T20:44:15.012 INFO:tasks.workunit.client.0.smithi139.stdout: "item": -1, 2024-09-15T20:44:15.012 INFO:tasks.workunit.client.0.smithi139.stdout: "item_name": "default" 2024-09-15T20:44:15.012 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.012 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.013 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "choose_indep", 2024-09-15T20:44:15.013 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 0, 2024-09-15T20:44:15.013 INFO:tasks.workunit.client.0.smithi139.stdout: "type": "osd" 2024-09-15T20:44:15.013 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.013 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.013 INFO:tasks.workunit.client.0.smithi139.stdout: "op": "emit" 2024-09-15T20:44:15.013 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:15.013 INFO:tasks.workunit.client.0.smithi139.stdout: ] 2024-09-15T20:44:15.013 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:15.013 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:15.013 INFO:tasks.workunit.client.0.smithi139.stdout: "tunables": { 2024-09-15T20:44:15.013 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_local_tries": 0, 2024-09-15T20:44:15.013 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_local_fallback_tries": 0, 2024-09-15T20:44:15.014 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_total_tries": 50, 2024-09-15T20:44:15.014 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_descend_once": 1, 2024-09-15T20:44:15.014 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_vary_r": 1, 2024-09-15T20:44:15.014 INFO:tasks.workunit.client.0.smithi139.stdout: "chooseleaf_stable": 1, 2024-09-15T20:44:15.014 INFO:tasks.workunit.client.0.smithi139.stdout: "msr_descents": 100, 2024-09-15T20:44:15.014 INFO:tasks.workunit.client.0.smithi139.stdout: "msr_collision_tries": 100, 2024-09-15T20:44:15.014 INFO:tasks.workunit.client.0.smithi139.stdout: "straw_calc_version": 1, 2024-09-15T20:44:15.014 INFO:tasks.workunit.client.0.smithi139.stdout: "allowed_bucket_algs": 54, 2024-09-15T20:44:15.014 INFO:tasks.workunit.client.0.smithi139.stdout: "profile": "jewel", 2024-09-15T20:44:15.014 INFO:tasks.workunit.client.0.smithi139.stdout: "optimal_tunables": 1, 2024-09-15T20:44:15.014 INFO:tasks.workunit.client.0.smithi139.stdout: "legacy_tunables": 0, 2024-09-15T20:44:15.014 INFO:tasks.workunit.client.0.smithi139.stdout: "minimum_required_version": "jewel", 2024-09-15T20:44:15.014 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables": 1, 2024-09-15T20:44:15.015 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables2": 1, 2024-09-15T20:44:15.015 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v2_rules": 1, 2024-09-15T20:44:15.015 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables3": 1, 2024-09-15T20:44:15.015 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v3_rules": 0, 2024-09-15T20:44:15.015 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v4_buckets": 1, 2024-09-15T20:44:15.015 INFO:tasks.workunit.client.0.smithi139.stdout: "require_feature_tunables5": 1, 2024-09-15T20:44:15.015 INFO:tasks.workunit.client.0.smithi139.stdout: "has_v5_rules": 0, 2024-09-15T20:44:15.015 INFO:tasks.workunit.client.0.smithi139.stdout: "has_msr_rules": 0 2024-09-15T20:44:15.015 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.015 INFO:tasks.workunit.client.0.smithi139.stdout: "choose_args": {} 2024-09-15T20:44:15.015 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.015 INFO:tasks.workunit.client.0.smithi139.stdout: "fsmap": { 2024-09-15T20:44:15.015 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 1, 2024-09-15T20:44:15.015 INFO:tasks.workunit.client.0.smithi139.stdout: "btime": "2024-09-15T20:39:25:184970+0000", 2024-09-15T20:44:15.015 INFO:tasks.workunit.client.0.smithi139.stdout: "default_fscid": -1, 2024-09-15T20:44:15.016 INFO:tasks.workunit.client.0.smithi139.stdout: "compat": { 2024-09-15T20:44:15.016 INFO:tasks.workunit.client.0.smithi139.stdout: "compat": {}, 2024-09-15T20:44:15.016 INFO:tasks.workunit.client.0.smithi139.stdout: "ro_compat": {}, 2024-09-15T20:44:15.016 INFO:tasks.workunit.client.0.smithi139.stdout: "incompat": { 2024-09-15T20:44:15.016 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_1": "base v0.20", 2024-09-15T20:44:15.016 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_2": "client writeable ranges", 2024-09-15T20:44:15.016 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_3": "default file layouts on dirs", 2024-09-15T20:44:15.016 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_4": "dir inode in separate object", 2024-09-15T20:44:15.016 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_5": "mds uses versioned encoding", 2024-09-15T20:44:15.016 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-15T20:44:15.016 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_8": "no anchor table", 2024-09-15T20:44:15.016 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_9": "file layout v2", 2024-09-15T20:44:15.016 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_10": "snaprealm v2", 2024-09-15T20:44:15.016 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_11": "minor log segments", 2024-09-15T20:44:15.017 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_12": "quiesce subvolumes" 2024-09-15T20:44:15.017 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:15.017 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.017 INFO:tasks.workunit.client.0.smithi139.stdout: "feature_flags": { 2024-09-15T20:44:15.017 INFO:tasks.workunit.client.0.smithi139.stdout: "enable_multiple": true, 2024-09-15T20:44:15.017 INFO:tasks.workunit.client.0.smithi139.stdout: "ever_enabled_multiple": true 2024-09-15T20:44:15.017 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.017 INFO:tasks.workunit.client.0.smithi139.stdout: "standbys": [], 2024-09-15T20:44:15.017 INFO:tasks.workunit.client.0.smithi139.stdout: "filesystems": [] 2024-09-15T20:44:15.017 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.018 INFO:tasks.workunit.client.0.smithi139.stdout: "mdsmap_first_committed": 1, 2024-09-15T20:44:15.018 INFO:tasks.workunit.client.0.smithi139.stdout: "mdsmap_last_committed": 1, 2024-09-15T20:44:15.018 INFO:tasks.workunit.client.0.smithi139.stdout: "auth": { 2024-09-15T20:44:15.018 INFO:tasks.workunit.client.0.smithi139.stdout: "first_committed": 1, 2024-09-15T20:44:15.018 INFO:tasks.workunit.client.0.smithi139.stdout: "last_committed": 5, 2024-09-15T20:44:15.018 INFO:tasks.workunit.client.0.smithi139.stdout: "num_secrets": 3 2024-09-15T20:44:15.018 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.018 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg": 1, 2024-09-15T20:44:15.018 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_active": 1, 2024-09-15T20:44:15.018 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_unknown": 0, 2024-09-15T20:44:15.018 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osd": 3, 2024-09-15T20:44:15.018 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_sum": { 2024-09-15T20:44:15.018 INFO:tasks.workunit.client.0.smithi139.stdout: "stat_sum": { 2024-09-15T20:44:15.018 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes": 62667, 2024-09-15T20:44:15.018 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects": 101, 2024-09-15T20:44:15.019 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_clones": 1, 2024-09-15T20:44:15.019 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_copies": 303, 2024-09-15T20:44:15.019 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T20:44:15.019 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing": 0, 2024-09-15T20:44:15.019 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_degraded": 101, 2024-09-15T20:44:15.019 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_misplaced": 0, 2024-09-15T20:44:15.019 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_unfound": 0, 2024-09-15T20:44:15.019 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_dirty": 101, 2024-09-15T20:44:15.019 INFO:tasks.workunit.client.0.smithi139.stdout: "num_whiteouts": 0, 2024-09-15T20:44:15.019 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read": 0, 2024-09-15T20:44:15.019 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read_kb": 0, 2024-09-15T20:44:15.019 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write": 101, 2024-09-15T20:44:15.019 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write_kb": 101, 2024-09-15T20:44:15.019 INFO:tasks.workunit.client.0.smithi139.stdout: "num_scrub_errors": 0, 2024-09-15T20:44:15.020 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T20:44:15.020 INFO:tasks.workunit.client.0.smithi139.stdout: "num_deep_scrub_errors": 0, 2024-09-15T20:44:15.020 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_recovered": 0, 2024-09-15T20:44:15.020 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_recovered": 0, 2024-09-15T20:44:15.020 INFO:tasks.workunit.client.0.smithi139.stdout: "num_keys_recovered": 0, 2024-09-15T20:44:15.020 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_omap": 0, 2024-09-15T20:44:15.020 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T20:44:15.020 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T20:44:15.020 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush": 0, 2024-09-15T20:44:15.020 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_kb": 0, 2024-09-15T20:44:15.020 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict": 0, 2024-09-15T20:44:15.020 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_kb": 0, 2024-09-15T20:44:15.020 INFO:tasks.workunit.client.0.smithi139.stdout: "num_promote": 0, 2024-09-15T20:44:15.021 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_high": 0, 2024-09-15T20:44:15.021 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_low": 0, 2024-09-15T20:44:15.021 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_some": 0, 2024-09-15T20:44:15.021 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_full": 0, 2024-09-15T20:44:15.021 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_pinned": 0, 2024-09-15T20:44:15.021 INFO:tasks.workunit.client.0.smithi139.stdout: "num_legacy_snapsets": 0, 2024-09-15T20:44:15.021 INFO:tasks.workunit.client.0.smithi139.stdout: "num_large_omap_objects": 0, 2024-09-15T20:44:15.021 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_manifest": 0, 2024-09-15T20:44:15.021 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_bytes": 0, 2024-09-15T20:44:15.021 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_keys": 0, 2024-09-15T20:44:15.021 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_repaired": 0 2024-09-15T20:44:15.021 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.021 INFO:tasks.workunit.client.0.smithi139.stdout: "store_stats": { 2024-09-15T20:44:15.021 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 0, 2024-09-15T20:44:15.022 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 0, 2024-09-15T20:44:15.022 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:44:15.022 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 0, 2024-09-15T20:44:15.022 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 0, 2024-09-15T20:44:15.022 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:44:15.022 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:44:15.022 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:44:15.022 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 0, 2024-09-15T20:44:15.022 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 0 2024-09-15T20:44:15.022 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.022 INFO:tasks.workunit.client.0.smithi139.stdout: "log_size": 18, 2024-09-15T20:44:15.022 INFO:tasks.workunit.client.0.smithi139.stdout: "ondisk_log_size": 18, 2024-09-15T20:44:15.022 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 3, 2024-09-15T20:44:15.022 INFO:tasks.workunit.client.0.smithi139.stdout: "acting": 3, 2024-09-15T20:44:15.023 INFO:tasks.workunit.client.0.smithi139.stdout: "num_store_stats": 0 2024-09-15T20:44:15.023 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.023 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_sum": { 2024-09-15T20:44:15.023 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 0, 2024-09-15T20:44:15.023 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 0, 2024-09-15T20:44:15.023 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pgs": 3, 2024-09-15T20:44:15.023 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osds": 3, 2024-09-15T20:44:15.023 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_osds": 3, 2024-09-15T20:44:15.023 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T20:44:15.023 INFO:tasks.workunit.client.0.smithi139.stdout: "kb": 314572800, 2024-09-15T20:44:15.023 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used": 85132, 2024-09-15T20:44:15.023 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_data": 2308, 2024-09-15T20:44:15.023 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_omap": 15, 2024-09-15T20:44:15.024 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_meta": 82736, 2024-09-15T20:44:15.024 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_avail": 314487668, 2024-09-15T20:44:15.024 INFO:tasks.workunit.client.0.smithi139.stdout: "statfs": { 2024-09-15T20:44:15.024 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 322122547200, 2024-09-15T20:44:15.024 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 322035372032, 2024-09-15T20:44:15.024 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:44:15.024 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 2363392, 2024-09-15T20:44:15.024 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 1611885, 2024-09-15T20:44:15.024 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:44:15.024 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:44:15.024 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:44:15.024 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 16263, 2024-09-15T20:44:15.024 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 84721785 2024-09-15T20:44:15.024 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.025 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_peers": [], 2024-09-15T20:44:15.025 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_trim_queue_len": 0, 2024-09-15T20:44:15.025 INFO:tasks.workunit.client.0.smithi139.stdout: "num_snap_trimming": 0, 2024-09-15T20:44:15.025 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shards_repaired": 0, 2024-09-15T20:44:15.025 INFO:tasks.workunit.client.0.smithi139.stdout: "op_queue_age_hist": { 2024-09-15T20:44:15.025 INFO:tasks.workunit.client.0.smithi139.stdout: "histogram": [], 2024-09-15T20:44:15.025 INFO:tasks.workunit.client.0.smithi139.stdout: "upper_bound": 1 2024-09-15T20:44:15.025 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.025 INFO:tasks.workunit.client.0.smithi139.stdout: "perf_stat": { 2024-09-15T20:44:15.025 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ms": 0, 2024-09-15T20:44:15.025 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ms": 0, 2024-09-15T20:44:15.025 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ns": 0, 2024-09-15T20:44:15.025 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ns": 0 2024-09-15T20:44:15.025 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.026 INFO:tasks.workunit.client.0.smithi139.stdout: "alerts": [], 2024-09-15T20:44:15.026 INFO:tasks.workunit.client.0.smithi139.stdout: "network_ping_times": [] 2024-09-15T20:44:15.026 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.026 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_sum_by_class": { 2024-09-15T20:44:15.026 INFO:tasks.workunit.client.0.smithi139.stdout: "hdd": { 2024-09-15T20:44:15.026 INFO:tasks.workunit.client.0.smithi139.stdout: "up_from": 0, 2024-09-15T20:44:15.026 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 0, 2024-09-15T20:44:15.026 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pgs": 3, 2024-09-15T20:44:15.026 INFO:tasks.workunit.client.0.smithi139.stdout: "num_osds": 3, 2024-09-15T20:44:15.026 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_osds": 3, 2024-09-15T20:44:15.026 INFO:tasks.workunit.client.0.smithi139.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T20:44:15.026 INFO:tasks.workunit.client.0.smithi139.stdout: "kb": 314572800, 2024-09-15T20:44:15.026 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used": 85132, 2024-09-15T20:44:15.026 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_data": 2308, 2024-09-15T20:44:15.027 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_omap": 15, 2024-09-15T20:44:15.027 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_used_meta": 82736, 2024-09-15T20:44:15.027 INFO:tasks.workunit.client.0.smithi139.stdout: "kb_avail": 314487668, 2024-09-15T20:44:15.027 INFO:tasks.workunit.client.0.smithi139.stdout: "statfs": { 2024-09-15T20:44:15.027 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 322122547200, 2024-09-15T20:44:15.027 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 322035372032, 2024-09-15T20:44:15.027 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:44:15.027 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 2363392, 2024-09-15T20:44:15.027 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 1611885, 2024-09-15T20:44:15.027 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:44:15.027 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:44:15.027 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:44:15.027 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 16263, 2024-09-15T20:44:15.027 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 84721785 2024-09-15T20:44:15.028 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.028 INFO:tasks.workunit.client.0.smithi139.stdout: "hb_peers": [], 2024-09-15T20:44:15.028 INFO:tasks.workunit.client.0.smithi139.stdout: "snap_trim_queue_len": 0, 2024-09-15T20:44:15.028 INFO:tasks.workunit.client.0.smithi139.stdout: "num_snap_trimming": 0, 2024-09-15T20:44:15.028 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shards_repaired": 0, 2024-09-15T20:44:15.028 INFO:tasks.workunit.client.0.smithi139.stdout: "op_queue_age_hist": { 2024-09-15T20:44:15.028 INFO:tasks.workunit.client.0.smithi139.stdout: "histogram": [], 2024-09-15T20:44:15.028 INFO:tasks.workunit.client.0.smithi139.stdout: "upper_bound": 1 2024-09-15T20:44:15.028 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.028 INFO:tasks.workunit.client.0.smithi139.stdout: "perf_stat": { 2024-09-15T20:44:15.028 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ms": 0, 2024-09-15T20:44:15.028 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ms": 0, 2024-09-15T20:44:15.028 INFO:tasks.workunit.client.0.smithi139.stdout: "commit_latency_ns": 0, 2024-09-15T20:44:15.028 INFO:tasks.workunit.client.0.smithi139.stdout: "apply_latency_ns": 0 2024-09-15T20:44:15.029 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.029 INFO:tasks.workunit.client.0.smithi139.stdout: "alerts": [], 2024-09-15T20:44:15.029 INFO:tasks.workunit.client.0.smithi139.stdout: "network_ping_times": [] 2024-09-15T20:44:15.029 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:15.029 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.029 INFO:tasks.workunit.client.0.smithi139.stdout: "pool_stats": [ 2024-09-15T20:44:15.029 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.029 INFO:tasks.workunit.client.0.smithi139.stdout: "poolid": 1, 2024-09-15T20:44:15.029 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg": 1, 2024-09-15T20:44:15.029 INFO:tasks.workunit.client.0.smithi139.stdout: "stat_sum": { 2024-09-15T20:44:15.029 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes": 62667, 2024-09-15T20:44:15.029 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects": 101, 2024-09-15T20:44:15.029 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_clones": 1, 2024-09-15T20:44:15.029 INFO:tasks.workunit.client.0.smithi139.stdout: "num_object_copies": 303, 2024-09-15T20:44:15.030 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T20:44:15.030 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_missing": 0, 2024-09-15T20:44:15.030 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_degraded": 101, 2024-09-15T20:44:15.030 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_misplaced": 0, 2024-09-15T20:44:15.030 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_unfound": 0, 2024-09-15T20:44:15.030 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_dirty": 101, 2024-09-15T20:44:15.030 INFO:tasks.workunit.client.0.smithi139.stdout: "num_whiteouts": 0, 2024-09-15T20:44:15.030 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read": 0, 2024-09-15T20:44:15.030 INFO:tasks.workunit.client.0.smithi139.stdout: "num_read_kb": 0, 2024-09-15T20:44:15.030 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write": 101, 2024-09-15T20:44:15.030 INFO:tasks.workunit.client.0.smithi139.stdout: "num_write_kb": 101, 2024-09-15T20:44:15.030 INFO:tasks.workunit.client.0.smithi139.stdout: "num_scrub_errors": 0, 2024-09-15T20:44:15.030 INFO:tasks.workunit.client.0.smithi139.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T20:44:15.030 INFO:tasks.workunit.client.0.smithi139.stdout: "num_deep_scrub_errors": 0, 2024-09-15T20:44:15.030 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_recovered": 0, 2024-09-15T20:44:15.030 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_recovered": 0, 2024-09-15T20:44:15.031 INFO:tasks.workunit.client.0.smithi139.stdout: "num_keys_recovered": 0, 2024-09-15T20:44:15.031 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_omap": 0, 2024-09-15T20:44:15.031 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T20:44:15.031 INFO:tasks.workunit.client.0.smithi139.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T20:44:15.031 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush": 0, 2024-09-15T20:44:15.031 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_kb": 0, 2024-09-15T20:44:15.031 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict": 0, 2024-09-15T20:44:15.031 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_kb": 0, 2024-09-15T20:44:15.031 INFO:tasks.workunit.client.0.smithi139.stdout: "num_promote": 0, 2024-09-15T20:44:15.031 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_high": 0, 2024-09-15T20:44:15.031 INFO:tasks.workunit.client.0.smithi139.stdout: "num_flush_mode_low": 0, 2024-09-15T20:44:15.031 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_some": 0, 2024-09-15T20:44:15.031 INFO:tasks.workunit.client.0.smithi139.stdout: "num_evict_mode_full": 0, 2024-09-15T20:44:15.031 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_pinned": 0, 2024-09-15T20:44:15.032 INFO:tasks.workunit.client.0.smithi139.stdout: "num_legacy_snapsets": 0, 2024-09-15T20:44:15.032 INFO:tasks.workunit.client.0.smithi139.stdout: "num_large_omap_objects": 0, 2024-09-15T20:44:15.032 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_manifest": 0, 2024-09-15T20:44:15.032 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_bytes": 0, 2024-09-15T20:44:15.032 INFO:tasks.workunit.client.0.smithi139.stdout: "num_omap_keys": 0, 2024-09-15T20:44:15.032 INFO:tasks.workunit.client.0.smithi139.stdout: "num_objects_repaired": 0 2024-09-15T20:44:15.032 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.032 INFO:tasks.workunit.client.0.smithi139.stdout: "store_stats": { 2024-09-15T20:44:15.032 INFO:tasks.workunit.client.0.smithi139.stdout: "total": 0, 2024-09-15T20:44:15.032 INFO:tasks.workunit.client.0.smithi139.stdout: "available": 0, 2024-09-15T20:44:15.032 INFO:tasks.workunit.client.0.smithi139.stdout: "internally_reserved": 0, 2024-09-15T20:44:15.033 INFO:tasks.workunit.client.0.smithi139.stdout: "allocated": 1241088, 2024-09-15T20:44:15.033 INFO:tasks.workunit.client.0.smithi139.stdout: "data_stored": 1241088, 2024-09-15T20:44:15.033 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed": 0, 2024-09-15T20:44:15.033 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_allocated": 0, 2024-09-15T20:44:15.033 INFO:tasks.workunit.client.0.smithi139.stdout: "data_compressed_original": 0, 2024-09-15T20:44:15.033 INFO:tasks.workunit.client.0.smithi139.stdout: "omap_allocated": 0, 2024-09-15T20:44:15.033 INFO:tasks.workunit.client.0.smithi139.stdout: "internal_metadata": 0 2024-09-15T20:44:15.033 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.033 INFO:tasks.workunit.client.0.smithi139.stdout: "log_size": 18, 2024-09-15T20:44:15.033 INFO:tasks.workunit.client.0.smithi139.stdout: "ondisk_log_size": 18, 2024-09-15T20:44:15.033 INFO:tasks.workunit.client.0.smithi139.stdout: "up": 3, 2024-09-15T20:44:15.033 INFO:tasks.workunit.client.0.smithi139.stdout: "acting": 3, 2024-09-15T20:44:15.033 INFO:tasks.workunit.client.0.smithi139.stdout: "num_store_stats": 3 2024-09-15T20:44:15.034 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:15.034 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:15.034 INFO:tasks.workunit.client.0.smithi139.stdout: "osd_stats": [ 2024-09-15T20:44:15.034 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.034 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:44:15.034 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 197568495627 2024-09-15T20:44:15.034 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.034 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.034 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:44:15.034 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 42949672993 2024-09-15T20:44:15.034 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.034 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.034 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:44:15.034 INFO:tasks.workunit.client.0.smithi139.stdout: "seq": 214748364810 2024-09-15T20:44:15.034 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:15.035 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:15.035 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_by_state": [ 2024-09-15T20:44:15.035 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.035 INFO:tasks.workunit.client.0.smithi139.stdout: "state": "active+undersized+degraded", 2024-09-15T20:44:15.035 INFO:tasks.workunit.client.0.smithi139.stdout: "num": 1 2024-09-15T20:44:15.035 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:15.035 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:15.035 INFO:tasks.workunit.client.0.smithi139.stdout: "num_pg_by_osd": [ 2024-09-15T20:44:15.035 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.035 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2147483647, 2024-09-15T20:44:15.035 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 0, 2024-09-15T20:44:15.035 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 1, 2024-09-15T20:44:15.035 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:44:15.035 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.035 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.036 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 2, 2024-09-15T20:44:15.036 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 0, 2024-09-15T20:44:15.036 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 1, 2024-09-15T20:44:15.036 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:44:15.036 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.036 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.036 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 0, 2024-09-15T20:44:15.036 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 1, 2024-09-15T20:44:15.036 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 1, 2024-09-15T20:44:15.036 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:44:15.036 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.036 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.036 INFO:tasks.workunit.client.0.smithi139.stdout: "osd": 1, 2024-09-15T20:44:15.037 INFO:tasks.workunit.client.0.smithi139.stdout: "num_primary_pg": 0, 2024-09-15T20:44:15.037 INFO:tasks.workunit.client.0.smithi139.stdout: "num_acting_pg": 0, 2024-09-15T20:44:15.037 INFO:tasks.workunit.client.0.smithi139.stdout: "num_up_not_acting_pg": 0 2024-09-15T20:44:15.037 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:15.037 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:15.037 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snaps": [ 2024-09-15T20:44:15.037 INFO:tasks.workunit.client.0.smithi139.stdout: { 2024-09-15T20:44:15.037 INFO:tasks.workunit.client.0.smithi139.stdout: "pool": 1, 2024-09-15T20:44:15.037 INFO:tasks.workunit.client.0.smithi139.stdout: "purged_snaps": {} 2024-09-15T20:44:15.037 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:15.037 INFO:tasks.workunit.client.0.smithi139.stdout: ], 2024-09-15T20:44:15.037 INFO:tasks.workunit.client.0.smithi139.stdout: "servicemap": { 2024-09-15T20:44:15.037 INFO:tasks.workunit.client.0.smithi139.stdout: "epoch": 5, 2024-09-15T20:44:15.037 INFO:tasks.workunit.client.0.smithi139.stdout: "modified": "2024-09-15T20:42:20.827915+0000", 2024-09-15T20:44:15.037 INFO:tasks.workunit.client.0.smithi139.stdout: "services": {} 2024-09-15T20:44:15.038 INFO:tasks.workunit.client.0.smithi139.stdout: }, 2024-09-15T20:44:15.038 INFO:tasks.workunit.client.0.smithi139.stdout: "mgrstat_first_committed": 1, 2024-09-15T20:44:15.038 INFO:tasks.workunit.client.0.smithi139.stdout: "mgrstat_last_committed": 133, 2024-09-15T20:44:15.038 INFO:tasks.workunit.client.0.smithi139.stdout: "logm_first_committed": 1, 2024-09-15T20:44:15.038 INFO:tasks.workunit.client.0.smithi139.stdout: "logm_last_committed": 509, 2024-09-15T20:44:15.038 INFO:tasks.workunit.client.0.smithi139.stdout: "paxos": { 2024-09-15T20:44:15.038 INFO:tasks.workunit.client.0.smithi139.stdout: "first_committed": 1, 2024-09-15T20:44:15.038 INFO:tasks.workunit.client.0.smithi139.stdout: "last_committed": 704, 2024-09-15T20:44:15.038 INFO:tasks.workunit.client.0.smithi139.stdout: "last_pn": 0, 2024-09-15T20:44:15.038 INFO:tasks.workunit.client.0.smithi139.stdout: "accepted_pn": 0 2024-09-15T20:44:15.038 INFO:tasks.workunit.client.0.smithi139.stdout: } 2024-09-15T20:44:15.038 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:44:15.039 INFO:tasks.workunit.client.0.smithi139.stderr:report 654997597 2024-09-15T20:44:15.039 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-15T20:44:15.039 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2024-09-15T20:44:15.039 INFO:tasks.workunit.client.0.smithi139.stdout:delay recovery 2024-09-15T20:44:15.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T20:44:15.370 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:44:15.370 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-15T20:43:35.174867+0000 43'102 52:143 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-15T20:41:23.916663+0000 0'0 2024-09-15T20:41:23.916663+0000 0 0 periodic scrub scheduled @ 2024-09-17T00:34:10.093200+0000 0 0 2024-09-15T20:44:15.371 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:44:15.371 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:44:15.371 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:44:15.386 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-09-15T20:44:15.387 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2024-09-15T20:44:15.387 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-15T20:44:15.387 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-15T20:44:15.388 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:44:15.388 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:15.388 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:44:15.388 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.0.asok 2024-09-15T20:44:15.389 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-09-15T20:44:15.389 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.0.asok set_recovery_delay 100000 2024-09-15T20:44:15.495 INFO:tasks.workunit.client.0.smithi139.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T20:44:15.506 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-09-15T20:44:15.507 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2024-09-15T20:44:15.507 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-15T20:44:15.507 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-15T20:44:15.508 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:44:15.508 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:15.508 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:44:15.508 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.2.asok 2024-09-15T20:44:15.509 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-09-15T20:44:15.509 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.2.asok set_recovery_delay 100000 2024-09-15T20:44:15.614 INFO:tasks.workunit.client.0.smithi139.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T20:44:15.626 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2024-09-15T20:44:15.626 INFO:tasks.workunit.client.0.smithi139.stdout:revive divergent 1 2024-09-15T20:44:15.626 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T20:44:16.049 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:44:16.050 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-15T20:43:35.174867+0000 43'102 52:143 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-15T20:41:23.916663+0000 0'0 2024-09-15T20:41:23.916663+0000 0 0 periodic scrub scheduled @ 2024-09-17T00:34:10.093200+0000 0 0 2024-09-15T20:44:16.050 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:44:16.050 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:44:16.050 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:44:16.065 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2024-09-15T20:44:16.803 INFO:tasks.workunit.client.0.smithi139.stderr:noup is set 2024-09-15T20:44:16.822 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-09-15T20:44:16.822 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-15T20:44:16.822 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:44:16.822 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:44:16.823 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:44:16.823 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-15T20:44:16.823 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:44:16.823 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:44:16.823 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:44:16.823 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:44:16.823 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T20:44:16.823 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T20:44:16.823 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:44:16.823 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:44:16.823 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:44:16.823 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:44:16.823 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:44:16.824 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:44:16.825 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:44:16.825 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:16.825 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:44:16.825 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:44:16.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:44:16.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:44:16.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:44:16.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:44:16.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:44:16.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:44:16.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:44:16.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:44:16.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:44:16.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:44:16.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-15T20:44:16.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:44:16.827 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.1 2024-09-15T20:44:16.827 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:44:16.827 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-15T20:44:16.827 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:44:16.828 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:44:16.828 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:44:16.828 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:44:16.867 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:44:16.868+0000 7ff76d8e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:16.879 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:44:16.880+0000 7ff76d8e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:16.880 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:44:16.880+0000 7ff76d8e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:17.273 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-15T20:44:17.273 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2024-09-15T20:44:18.152 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:44:18.152+0000 7ff76d8e77c0 -1 Falling back to public interface 2024-09-15T20:44:20.874 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:44:20.876+0000 7ff76d8e77c0 -1 osd.1 30 log_to_monitors true 2024-09-15T20:44:22.274 INFO:tasks.workunit.client.0.smithi139.stdout:delay recovery divergent 2024-09-15T20:44:22.274 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2024-09-15T20:44:22.274 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T20:44:22.695 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:44:22.696 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-15T20:43:35.174867+0000 43'102 54:147 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-15T20:41:23.916663+0000 0'0 2024-09-15T20:41:23.916663+0000 0 0 periodic scrub scheduled @ 2024-09-17T00:34:10.093200+0000 0 0 2024-09-15T20:44:22.696 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:44:22.696 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:44:22.696 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:44:22.712 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2024-09-15T20:44:22.712 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-15T20:44:22.712 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-15T20:44:22.713 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:44:22.713 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:22.713 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:44:22.714 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27419/ceph-osd.1.asok 2024-09-15T20:44:22.714 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2024-09-15T20:44:22.714 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27419/ceph-osd.1.asok set_recovery_delay 100000 2024-09-15T20:44:22.829 INFO:tasks.workunit.client.0.smithi139.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T20:44:22.842 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2024-09-15T20:44:23.631 INFO:tasks.workunit.client.0.smithi139.stderr:noup is unset 2024-09-15T20:44:23.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2024-09-15T20:44:23.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:44:23.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:44:23.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:44:23.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:44:23.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:23.651 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:44:23.652 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:44:23.652 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:23.652 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:44:24.218 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 46 up_thru 51 down_at 45 last_clean_interval [37,43) [v2:127.0.0.1:6802/463017228,v1:127.0.0.1:6803/463017228] [v2:127.0.0.1:6804/463017228,v1:127.0.0.1:6805/463017228] exists,up 072ef209-0424-4dcb-ba88-59a19eaa46bd 2024-09-15T20:44:24.219 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:44:24.219 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:44:24.219 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:44:24.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2024-09-15T20:44:24.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:44:24.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:44:24.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:44:24.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:44:24.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:24.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:44:24.220 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:44:24.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:24.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:44:24.699 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/1718148900,v1:127.0.0.1:6819/1718148900] [v2:127.0.0.1:6820/1718148900,v1:127.0.0.1:6821/1718148900] exists,up 61477bed-b55a-47e6-8ac3-ac4772590dce 2024-09-15T20:44:24.699 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:44:24.700 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:44:24.700 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:44:24.700 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2024-09-15T20:44:24.700 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:44:24.700 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:44:24.700 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:44:24.700 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:44:24.700 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:24.700 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:44:24.700 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:44:24.701 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:24.701 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:44:25.170 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 49 last_clean_interval [39,43) [v2:127.0.0.1:6810/430831352,v1:127.0.0.1:6811/430831352] [v2:127.0.0.1:6812/430831352,v1:127.0.0.1:6813/430831352] exists,up e0a289bd-3c1f-48e6-b612-9a2c75ec33bb 2024-09-15T20:44:25.170 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:44:25.171 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:44:25.171 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:44:25.171 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T20:44:25.573 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:44:25.573 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-15T20:43:35.174867+0000 43'102 56:151 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-15T20:41:23.916663+0000 0'0 2024-09-15T20:41:23.916663+0000 0 0 periodic scrub scheduled @ 2024-09-17T00:34:10.093200+0000 0 0 2024-09-15T20:44:25.573 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:44:25.574 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:44:25.574 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:44:25.589 INFO:tasks.workunit.client.0.smithi139.stdout:wait for peering 2024-09-15T20:44:25.589 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2024-09-15T20:44:25.589 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T20:44:25.985 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:44:25.985 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-15T20:43:35.174867+0000 43'102 56:151 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-15T20:41:23.916663+0000 0'0 2024-09-15T20:41:23.916663+0000 0 0 periodic scrub scheduled @ 2024-09-17T00:34:10.093200+0000 0 0 2024-09-15T20:44:25.985 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:44:25.985 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:44:25.985 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:44:26.000 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2024-09-15T20:44:26.160 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2024-09-15T20:44:26.160 INFO:tasks.workunit.client.0.smithi139.stdout:killing divergent 1 2024-09-15T20:44:26.160 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T20:44:26.633 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:44:26.633 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-15T20:43:35.174867+0000 43'102 56:151 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-15T20:41:23.916663+0000 0'0 2024-09-15T20:41:23.916663+0000 0 0 periodic scrub scheduled @ 2024-09-17T00:34:10.093200+0000 0 0 2024-09-15T20:44:26.633 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:44:26.633 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:44:26.633 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:44:26.649 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2024-09-15T20:44:26.650 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:44:26.650 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:44:26.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:44:26.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:44:26.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:44:26.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:44:26.759 INFO:tasks.workunit.client.0.smithi139.stdout:reviving divergent 1 2024-09-15T20:44:26.760 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2024-09-15T20:44:26.760 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T20:44:27.174 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:44:27.174 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 101 4 4 0 0 62667 0 0 12 90 12 active+recovering+degraded 2024-09-15T20:44:25.094771+0000 43'102 59:136 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:41:23.916663+0000 0'0 2024-09-15T20:41:23.916663+0000 0 0 no scrub is scheduled 0 0 2024-09-15T20:44:27.174 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:44:27.175 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:44:27.175 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:44:27.191 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-09-15T20:44:27.191 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-15T20:44:27.191 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:44:27.191 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:44:27.191 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:44:27.191 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-15T20:44:27.191 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T20:44:27.191 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:44:27.192 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:44:27.192 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:44:27.192 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T20:44:27.192 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T20:44:27.192 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:44:27.192 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:44:27.192 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T20:44:27.192 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:44:27.192 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:44:27.192 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:44:27.194 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:44:27.194 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:27.194 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:44:27.194 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:44:27.194 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' 2024-09-15T20:44:27.194 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:44:27.194 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T20:44:27.194 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T20:44:27.194 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:44:27.194 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:44:27.194 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:44:27.194 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:44:27.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:44:27.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:44:27.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-15T20:44:27.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:44:27.195 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.1 2024-09-15T20:44:27.196 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0e220df1-ac9c-433e-8885-cf9f7aab293e --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27419/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:44:27.196 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-15T20:44:27.196 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:44:27.196 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:44:27.196 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:44:27.196 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:44:27.233 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:44:27.236+0000 7fed40eda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:27.250 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:44:27.252+0000 7fed40eda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:27.251 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:44:27.252+0000 7fed40eda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:27.731 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:44:27.731 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:44:27.732 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:44:27.732 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:44:27.732 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:44:27.732 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:44:27.732 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:44:27.732 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:44:27.733 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:44:27.733 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:44:28.177 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/1718148900,v1:127.0.0.1:6819/1718148900] [v2:127.0.0.1:6820/1718148900,v1:127.0.0.1:6821/1718148900] exists,up 61477bed-b55a-47e6-8ac3-ac4772590dce 2024-09-15T20:44:28.178 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:44:28.178 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:44:28.178 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:44:28.178 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2024-09-15T20:44:28.520 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:44:28.519+0000 7fed40eda7c0 -1 Falling back to public interface 2024-09-15T20:44:31.827 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:44:31.827+0000 7fed40eda7c0 -1 osd.1 59 log_to_monitors true 2024-09-15T20:44:48.180 INFO:tasks.workunit.client.0.smithi139.stdout:allowing recovery 2024-09-15T20:44:48.181 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2024-09-15T20:44:48.181 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T20:44:48.604 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:44:48.605 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-09-15T20:44:34.083284+0000 59'103 64:161 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:41:23.916663+0000 0'0 2024-09-15T20:41:23.916663+0000 0 0 periodic scrub scheduled @ 2024-09-17T06:46:00.511706+0000 0 0 2024-09-15T20:44:48.605 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:44:48.605 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:44:48.605 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:44:48.620 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-15T20:44:48.620 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-15T20:44:48.772 INFO:tasks.workunit.client.0.smithi139.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T20:44:48.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-15T20:44:48.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-15T20:44:48.936 INFO:tasks.workunit.client.0.smithi139.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T20:44:48.950 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-15T20:44:48.950 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-15T20:44:49.094 INFO:tasks.workunit.client.0.smithi139.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T20:44:49.107 INFO:tasks.workunit.client.0.smithi139.stdout:reading divergent objects 2024-09-15T20:44:49.107 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2024-09-15T20:44:49.107 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T20:44:49.493 INFO:tasks.workunit.client.0.smithi139.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T20:44:49.493 INFO:tasks.workunit.client.0.smithi139.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-09-15T20:44:34.083284+0000 59'103 64:161 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T20:41:23.916663+0000 0'0 2024-09-15T20:41:23.916663+0000 0 0 periodic scrub scheduled @ 2024-09-17T06:46:00.511706+0000 0 0 2024-09-15T20:44:49.493 INFO:tasks.workunit.client.0.smithi139.stdout: 2024-09-15T20:44:49.493 INFO:tasks.workunit.client.0.smithi139.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T20:44:49.493 INFO:tasks.workunit.client.0.smithi139.stderr:dumped pgs 2024-09-15T20:44:49.509 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2024-09-15T20:44:49.510 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2024-09-15T20:44:49.511 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T20:44:49.511 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2024-09-15T20:44:49.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T20:44:49.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2024-09-15T20:44:49.598 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T20:44:49.598 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2024-09-15T20:44:49.633 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T20:44:49.633 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2024-09-15T20:44:49.668 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2024-09-15T20:44:49.669 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-09-15T20:44:49.670 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-15T20:44:49.675 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-09-15T20:42:04.244152+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-15T20:44:49.676 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 20 _merge_object_divergent_entries: keeping 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-09-15T20:42:04.244152+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-15T20:44:49.676 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T20:44:49.676 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 29'99 first_divergent_update: 30'103 last_divergent_update: 30'103 2024-09-15T20:44:49.676 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2024-09-15T20:44:49.676 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2024-09-15T20:44:49.676 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4659.0:1 2024-09-15T20:42:12.485260+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-09-15T20:44:49.676 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 20 _merge_object_divergent_entries: keeping 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4659.0:1 2024-09-15T20:42:12.485260+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-09-15T20:44:49.676 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T20:44:49.676 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 29'99 first_divergent_update: 30'104 last_divergent_update: 30'104 2024-09-15T20:44:49.676 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2024-09-15T20:44:49.677 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2024-09-15T20:44:49.677 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 30'104 2024-09-15T20:44:49.677 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4659.0:1 2024-09-15T20:42:12.485260+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-09-15T20:44:49.677 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2024-09-15T20:44:49.677 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4650.0:1 2024-09-15T20:42:11.239439+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false,29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4655.0:1 2024-09-15T20:42:12.247268+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-15T20:44:49.678 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4650.0:1 2024-09-15T20:42:11.239439+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-15T20:44:49.678 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 20 _merge_object_divergent_entries: keeping 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4655.0:1 2024-09-15T20:42:12.247268+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-15T20:44:49.678 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T20:44:49.678 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'102 2024-09-15T20:44:49.678 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-09-15T20:44:49.678 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-09-15T20:44:49.678 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 30'104 2024-09-15T20:44:49.678 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4655.0:1 2024-09-15T20:42:12.247268+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-15T20:44:49.678 INFO:tasks.workunit.client.0.smithi139.stdout:td/divergent-priors/osd.1.log:2024-09-15T20:44:25.008+0000 7ff751871640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-09-15T20:44:49.678 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-09-15T20:44:49.680 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-15T20:44:49.685 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2024-09-15T20:44:49.686 INFO:tasks.workunit.client.0.smithi139.stdout:success 2024-09-15T20:44:49.686 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2024-09-15T20:44:49.686 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-15T20:44:49.686 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-15T20:44:50.228 INFO:tasks.workunit.client.0.smithi139.stderr:pool 'test' does not exist 2024-09-15T20:44:50.245 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2024-09-15T20:44:50.245 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:44:50.246 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:44:50.246 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:44:50.246 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:44:50.246 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:44:55.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:44:55.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-15T20:44:55.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-15T20:44:55.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:44:55.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-15T20:44:55.565 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:44:55.565 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:44:55.566 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:44:55.566 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:44:55.566 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:44:55.575 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:44:55.575 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:44:55.576 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:44:55.576 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:44:55.577 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:44:55.577 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:44:55.578 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:44:55.579 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:44:55.579 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:44:55.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:44:55.580 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:44:55.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:44:55.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:44:55.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-15T20:44:55.609 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:44:55.609 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:55.609 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:44:55.609 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27419 2024-09-15T20:44:55.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:44:55.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:44:55.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-15T20:44:55.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/divergent-priors 0 2024-09-15T20:44:55.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-15T20:44:55.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-15T20:44:55.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-15T20:44:55.611 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:44:55.611 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:44:55.611 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:44:55.611 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:44:55.611 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:44:55.613 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:44:55.619 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:44:55.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:44:55.619 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:44:55.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:44:55.620 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:44:55.620 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:44:55.620 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:44:55.620 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:44:55.620 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:44:55.620 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:44:55.620 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:44:55.620 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T20:44:55.620 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-15T20:44:55.621 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:44:55.621 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:55.621 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27419 2024-09-15T20:44:55.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27419 2024-09-15T20:44:55.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:44:55.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:44:55.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-15T20:44:55.623 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T20:44:55.623 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T20:44:55.677 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2024-09-15T20:44:55.678 DEBUG:teuthology.orchestra.run.smithi139:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2024-09-15T20:44:55.734 INFO:tasks.workunit.client.0.smithi139.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T20:44:55.739 INFO:tasks.workunit.client.0.smithi139.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T20:44:55.739 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-15T20:44:55.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-15T20:44:55.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-15T20:44:55.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-15T20:44:55.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-15T20:44:55.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-15T20:44:55.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-15T20:44:55.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-15T20:44:55.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/ec-error-rollforward 2024-09-15T20:44:55.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2024-09-15T20:44:55.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2024-09-15T20:44:55.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2024-09-15T20:44:55.741 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2024-09-15T20:44:55.741 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2024-09-15T20:44:55.741 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2024-09-15T20:44:55.742 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none ' 2024-09-15T20:44:55.742 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2024-09-15T20:44:55.742 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2024-09-15T20:44:55.742 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2024-09-15T20:44:55.742 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2024-09-15T20:44:55.742 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2024-09-15T20:44:55.742 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2024-09-15T20:44:55.742 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2024-09-15T20:44:55.742 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T20:44:55.742 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2024-09-15T20:44:55.744 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2024-09-15T20:44:55.744 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2024-09-15T20:44:55.744 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2024-09-15T20:44:55.744 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ec-error-rollforward 2024-09-15T20:44:55.744 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ec-error-rollforward 2024-09-15T20:44:55.744 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ec-error-rollforward 2024-09-15T20:44:55.744 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:44:55.744 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-15T20:44:55.744 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:44:55.744 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:44:55.745 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:44:55.745 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:44:55.745 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:44:55.745 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:44:55.746 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:44:55.746 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:44:55.747 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:44:55.748 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:44:55.748 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:44:55.748 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:44:55.749 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:44:55.749 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:44:55.749 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:44:55.750 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:44:55.750 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:44:55.751 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:44:55.752 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ec-error-rollforward 2024-09-15T20:44:55.753 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:44:55.753 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:55.753 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.84938 2024-09-15T20:44:55.753 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.84938 2024-09-15T20:44:55.754 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:44:55.754 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:44:55.754 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ec-error-rollforward 2024-09-15T20:44:55.756 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:44:55.756 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:55.756 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.84938 2024-09-15T20:44:55.756 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.84938 2024-09-15T20:44:55.757 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:44:55.758 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-15T20:44:55.758 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-15T20:44:55.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:44:55.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2024-09-15T20:44:55.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2024-09-15T20:44:55.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2024-09-15T20:44:55.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2024-09-15T20:44:55.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ec-error-rollforward 2024-09-15T20:44:55.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T20:44:55.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T20:44:55.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T20:44:55.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ec-error-rollforward/a 2024-09-15T20:44:55.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2024-09-15T20:44:56.240 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T20:44:56.240 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:44:56.240 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:44:56.240 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:44:56.240 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:56.240 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.84938 2024-09-15T20:44:56.240 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.84938/$cluster-$name.asok' 2024-09-15T20:44:56.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T20:44:56.451 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T20:44:56.452 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T20:44:56.452 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:44:56.452 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:44:56.452 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T20:44:56.453 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T20:44:56.453 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:44:56.453 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:44:56.453 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:44:56.453 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:44:56.453 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:56.453 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.84938 2024-09-15T20:44:56.454 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.84938/ceph-mon.a.asok 2024-09-15T20:44:56.454 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:44:56.454 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.84938/ceph-mon.a.asok config get fsid 2024-09-15T20:44:56.571 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T20:44:56.571 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:44:56.571 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:44:56.571 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T20:44:56.572 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T20:44:56.572 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:44:56.572 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:44:56.572 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:44:56.572 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:44:56.572 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:56.572 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.84938 2024-09-15T20:44:56.573 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.84938/ceph-mon.a.asok 2024-09-15T20:44:56.573 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:44:56.573 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.84938/ceph-mon.a.asok config get mon_host 2024-09-15T20:44:56.682 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2024-09-15T20:44:56.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ec-error-rollforward 2024-09-15T20:44:56.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T20:44:56.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T20:44:56.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T20:44:56.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ec-error-rollforward/x 2024-09-15T20:44:56.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T20:44:57.579 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T20:44:57.579 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:44:57.580 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:44:57.580 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:44:57.580 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:57.581 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.84938 2024-09-15T20:44:57.581 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.84938/$cluster-$name.asok' 2024-09-15T20:44:57.581 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:44:57.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:44:57.618 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2024-09-15T20:44:57.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-15T20:44:57.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:44:57.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T20:44:57.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:44:57.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/0 2024-09-15T20:44:57.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-15T20:44:57.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:44:57.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:44:57.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:44:57.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-09-15T20:44:57.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-09-15T20:44:57.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:44:57.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:44:57.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-15T20:44:57.622 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:44:57.622 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:44:57.622 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:44:57.622 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:44:57.623 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:44:57.623 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.84938 2024-09-15T20:44:57.623 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.84938/$cluster-$name.asok' 2024-09-15T20:44:57.623 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' 2024-09-15T20:44:57.623 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:44:57.623 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:44:57.623 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:44:57.623 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-15T20:44:57.624 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-15T20:44:57.624 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:44:57.624 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:44:57.624 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:44:57.624 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:44:57.625 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:44:57.625 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:44:57.625 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/0 2024-09-15T20:44:57.625 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:44:57.626 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=46ae8d2d-04be-48da-a686-147d13c7798e 2024-09-15T20:44:57.626 INFO:tasks.workunit.client.0.smithi139.stdout:add osd0 46ae8d2d-04be-48da-a686-147d13c7798e 2024-09-15T20:44:57.626 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 46ae8d2d-04be-48da-a686-147d13c7798e' 2024-09-15T20:44:57.627 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:44:57.634 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBJR+dmohD7JRAAkxTCDq0GVQYn00DT2EnQrA== 2024-09-15T20:44:57.635 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBJR+dmohD7JRAAkxTCDq0GVQYn00DT2EnQrA=="}' 2024-09-15T20:44:57.635 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 46ae8d2d-04be-48da-a686-147d13c7798e -i td/ec-error-rollforward/0/new.json 2024-09-15T20:44:57.929 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:44:57.944 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/0/new.json 2024-09-15T20:44:57.945 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBJR+dmohD7JRAAkxTCDq0GVQYn00DT2EnQrA== --osd-uuid 46ae8d2d-04be-48da-a686-147d13c7798e 2024-09-15T20:44:57.981 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:44:57.978+0000 7f01a6fc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:57.998 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:44:57.994+0000 7f01a6fc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:57.999 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:44:57.998+0000 7f01a6fc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:44:57.999 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:44:57.998+0000 7f01a6fc97c0 -1 bdev(0x55dd29a89c00 td/ec-error-rollforward/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:44:58.000 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:44:57.998+0000 7f01a6fc97c0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2024-09-15T20:45:04.278 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2024-09-15T20:45:04.278 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:45:04.279 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd0 key to auth repository 2024-09-15T20:45:04.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T20:45:04.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:45:04.847 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T20:45:04.848 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T20:45:04.848 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:45:04.848 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:45:04.848 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:45:04.848 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:45:04.877 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:45:04.878+0000 7fb8a2bb87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:45:04.913 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:45:04.914+0000 7fb8a2bb87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:45:04.915 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:45:04.918+0000 7fb8a2bb87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:45:05.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T20:45:05.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:45:05.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:45:05.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:45:05.279 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:45:05.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:45:05.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:05.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:45:05.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:05.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:45:05.734 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:06.485 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:45:06.486+0000 7fb8a2bb87c0 -1 Falling back to public interface 2024-09-15T20:45:06.736 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:06.736 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:06.736 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:45:06.736 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:45:06.736 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:06.737 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:45:07.202 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:08.203 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:08.203 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:08.204 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:45:08.204 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:45:08.204 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:08.204 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:45:08.635 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:09.402 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:45:09.402+0000 7fb8a2bb87c0 -1 osd.0 0 log_to_monitors true 2024-09-15T20:45:09.637 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:09.637 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:09.638 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:45:09.638 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:45:09.638 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:09.638 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:45:10.208 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:11.209 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:11.209 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:11.209 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:45:11.210 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:45:11.210 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:11.210 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:45:11.680 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:12.681 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:12.681 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:12.681 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:45:12.682 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:45:12.682 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:12.682 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:45:13.225 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:14.227 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:14.228 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:14.228 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:45:14.228 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:45:14.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:14.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:45:14.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:15.653 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:15.653 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:15.653 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:45:15.654 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:45:15.654 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:15.654 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:45:16.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:17.225 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:17.225 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:17.225 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:45:17.226 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:45:17.226 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:17.226 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:45:17.658 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:18.660 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:18.660 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:18.660 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:45:18.661 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:45:18.661 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:18.661 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:45:19.218 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:20.219 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:20.219 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:20.220 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:45:20.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:45:20.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:20.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:45:20.670 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:21.671 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:21.671 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:21.672 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:45:21.672 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:45:21.672 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:21.672 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:45:22.295 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:23.297 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:23.297 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:23.297 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:45:23.298 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:45:23.298 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:23.298 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:45:24.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:25.062 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:25.062 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:25.063 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:45:25.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:45:25.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:25.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:45:25.760 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:26.762 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:26.762 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:26.762 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:45:26.763 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:45:26.763 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:26.763 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:45:27.855 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:28.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:28.857 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:28.857 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:45:28.857 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:45:28.857 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:28.857 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:45:29.989 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:30.990 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:45:30.991 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:30.991 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:30.991 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:45:30.991 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:30.991 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:45:32.081 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:33.082 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:33.082 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:33.082 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:45:33.083 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:45:33.083 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:33.083 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:45:33.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:34.624 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:34.624 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:34.624 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:45:34.624 INFO:tasks.workunit.client.0.smithi139.stdout:18 2024-09-15T20:45:34.624 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:34.625 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:45:35.114 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3721910055,v1:127.0.0.1:6803/3721910055] [v2:127.0.0.1:6804/3721910055,v1:127.0.0.1:6805/3721910055] exists,up 46ae8d2d-04be-48da-a686-147d13c7798e 2024-09-15T20:45:35.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:45:35.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:45:35.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:45:35.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2024-09-15T20:45:35.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-15T20:45:35.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:45:35.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T20:45:35.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:45:35.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/1 2024-09-15T20:45:35.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-15T20:45:35.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:45:35.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:45:35.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:45:35.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-09-15T20:45:35.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-09-15T20:45:35.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:45:35.122 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:45:35.123 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-15T20:45:35.123 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:45:35.123 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:45:35.123 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:45:35.123 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:45:35.123 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:45:35.123 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.84938 2024-09-15T20:45:35.123 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.84938/$cluster-$name.asok' 2024-09-15T20:45:35.123 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' 2024-09-15T20:45:35.124 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:45:35.124 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:45:35.124 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:45:35.124 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-15T20:45:35.124 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-15T20:45:35.124 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:45:35.124 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:45:35.124 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:45:35.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:45:35.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:45:35.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:45:35.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/1 2024-09-15T20:45:35.125 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:45:35.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=375b6031-d027-47c1-bddf-0d15d664926c 2024-09-15T20:45:35.126 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 375b6031-d027-47c1-bddf-0d15d664926c' 2024-09-15T20:45:35.126 INFO:tasks.workunit.client.0.smithi139.stdout:add osd1 375b6031-d027-47c1-bddf-0d15d664926c 2024-09-15T20:45:35.126 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:45:35.135 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBvR+dmR4w7CBAAZMiTmhwvNdJ5ftxNiMjzxg== 2024-09-15T20:45:35.135 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBvR+dmR4w7CBAAZMiTmhwvNdJ5ftxNiMjzxg=="}' 2024-09-15T20:45:35.135 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 375b6031-d027-47c1-bddf-0d15d664926c -i td/ec-error-rollforward/1/new.json 2024-09-15T20:45:35.628 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:45:35.644 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/1/new.json 2024-09-15T20:45:35.645 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBvR+dmR4w7CBAAZMiTmhwvNdJ5ftxNiMjzxg== --osd-uuid 375b6031-d027-47c1-bddf-0d15d664926c 2024-09-15T20:45:35.677 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:45:35.677+0000 7f7a701da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:45:35.680 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:45:35.681+0000 7f7a701da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:45:35.681 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:45:35.681+0000 7f7a701da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:45:35.682 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:45:35.681+0000 7f7a701da7c0 -1 bdev(0x5617e945dc00 td/ec-error-rollforward/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:45:35.682 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:45:35.685+0000 7f7a701da7c0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2024-09-15T20:45:42.745 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2024-09-15T20:45:42.746 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:45:42.747 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd1 key to auth repository 2024-09-15T20:45:42.748 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T20:45:42.748 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:45:43.315 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T20:45:43.315 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.1 2024-09-15T20:45:43.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:45:43.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:45:43.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:45:43.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:45:43.345 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:45:43.345+0000 7f7d909b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:45:43.366 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:45:43.365+0000 7f7d909b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:45:43.367 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:45:43.369+0000 7f7d909b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:45:43.739 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T20:45:43.739 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:45:43.739 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:45:43.739 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:45:43.740 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:45:43.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:45:43.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:43.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:45:43.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:43.741 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:44.156 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:44.633 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:45:44.633+0000 7f7d909b27c0 -1 Falling back to public interface 2024-09-15T20:45:45.158 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:45.158 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:45.158 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:45:45.159 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:45:45.159 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:45.159 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:45.596 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:46.597 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:46.598 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:46.598 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:45:46.598 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:45:46.598 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:46.599 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:47.021 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:47.542 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:45:47.541+0000 7f7d909b27c0 -1 osd.1 0 log_to_monitors true 2024-09-15T20:45:48.022 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:48.023 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:48.023 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:45:48.024 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:45:48.024 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:48.024 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:48.730 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:49.732 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:49.732 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:49.732 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:45:49.733 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:45:49.733 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:49.733 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:50.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:51.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:51.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:51.201 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:45:51.202 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:45:51.202 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:51.202 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:51.669 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:52.671 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:52.671 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:52.672 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:45:52.672 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:45:52.673 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:52.673 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:53.142 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:54.143 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:54.144 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:54.144 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:45:54.144 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:45:54.144 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:54.145 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:54.812 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:55.813 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:55.813 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:55.813 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:45:55.814 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:45:55.814 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:55.814 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:56.422 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:57.424 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:57.424 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:57.424 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:45:57.425 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:45:57.425 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:57.425 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:57.931 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:45:58.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:45:58.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:45:58.934 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:45:58.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:45:58.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:45:58.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:45:59.509 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:00.511 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:00.511 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:00.511 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:46:00.512 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:46:00.512 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:00.512 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:46:01.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:02.036 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:02.036 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:02.037 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:46:02.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:46:02.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:02.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:46:03.123 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:04.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:04.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:04.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:46:04.126 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:46:04.126 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:04.126 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:46:04.916 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:05.917 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:05.920 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:05.920 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:46:05.920 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:46:05.921 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:05.921 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:46:07.058 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:08.059 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:08.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:08.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:46:08.060 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:46:08.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:08.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:46:09.059 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:10.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:10.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:10.060 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:46:10.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:46:10.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:10.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:46:10.925 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:11.926 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:11.927 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:11.927 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:46:11.927 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:46:11.927 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:11.927 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:46:12.924 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:13.926 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:13.926 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:13.926 INFO:tasks.workunit.client.0.smithi139.stdout:18 2024-09-15T20:46:13.927 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:46:13.927 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:13.927 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:46:14.413 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2546374193,v1:127.0.0.1:6811/2546374193] [v2:127.0.0.1:6812/2546374193,v1:127.0.0.1:6813/2546374193] exists,up 375b6031-d027-47c1-bddf-0d15d664926c 2024-09-15T20:46:14.416 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:46:14.416 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:46:14.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:46:14.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2024-09-15T20:46:14.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-15T20:46:14.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:46:14.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T20:46:14.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:46:14.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/2 2024-09-15T20:46:14.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-15T20:46:14.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:46:14.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:46:14.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:46:14.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-09-15T20:46:14.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-09-15T20:46:14.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:46:14.418 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:46:14.418 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-15T20:46:14.418 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:46:14.418 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:46:14.418 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:46:14.418 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:46:14.418 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:46:14.418 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.84938 2024-09-15T20:46:14.418 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.84938/$cluster-$name.asok' 2024-09-15T20:46:14.418 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' 2024-09-15T20:46:14.418 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:46:14.418 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:46:14.418 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:46:14.418 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-15T20:46:14.419 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-15T20:46:14.419 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:46:14.419 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:46:14.419 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:46:14.419 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:46:14.419 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:46:14.419 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:46:14.419 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/2 2024-09-15T20:46:14.420 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:46:14.420 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=37bcd18d-fac2-4f1c-92a0-af6c822e2dd4 2024-09-15T20:46:14.420 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 37bcd18d-fac2-4f1c-92a0-af6c822e2dd4' 2024-09-15T20:46:14.420 INFO:tasks.workunit.client.0.smithi139.stdout:add osd2 37bcd18d-fac2-4f1c-92a0-af6c822e2dd4 2024-09-15T20:46:14.420 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:46:14.436 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCWR+dmxbUcGhAAD9/lHrNSpynZ7Pa1kzns1A== 2024-09-15T20:46:14.436 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCWR+dmxbUcGhAAD9/lHrNSpynZ7Pa1kzns1A=="}' 2024-09-15T20:46:14.436 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 37bcd18d-fac2-4f1c-92a0-af6c822e2dd4 -i td/ec-error-rollforward/2/new.json 2024-09-15T20:46:14.963 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:46:14.979 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/2/new.json 2024-09-15T20:46:14.980 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCWR+dmxbUcGhAAD9/lHrNSpynZ7Pa1kzns1A== --osd-uuid 37bcd18d-fac2-4f1c-92a0-af6c822e2dd4 2024-09-15T20:46:15.011 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:46:15.012+0000 7fac721387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:46:15.042 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:46:15.040+0000 7fac721387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:46:15.044 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:46:15.044+0000 7fac721387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:46:15.044 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:46:15.044+0000 7fac721387c0 -1 bdev(0x55d2cbcbfc00 td/ec-error-rollforward/2/block) open stat got: (1) Operation not permitted 2024-09-15T20:46:15.044 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:46:15.044+0000 7fac721387c0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2024-09-15T20:46:21.824 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2024-09-15T20:46:21.824 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:46:21.825 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd2 key to auth repository 2024-09-15T20:46:21.825 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T20:46:21.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:46:22.412 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T20:46:22.413 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.2 2024-09-15T20:46:22.413 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:46:22.413 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:46:22.413 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:46:22.413 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:46:22.443 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:46:22.443+0000 7faf5bac37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:46:22.462 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:46:22.459+0000 7faf5bac37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:46:22.463 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:46:22.463+0000 7faf5bac37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:46:22.858 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T20:46:22.858 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:46:22.858 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:46:22.858 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:46:22.858 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:46:22.858 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:22.858 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:46:22.858 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:46:22.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:22.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:23.277 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:23.742 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:46:23.739+0000 7faf5bac37c0 -1 Falling back to public interface 2024-09-15T20:46:24.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:24.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:24.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:46:24.280 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:46:24.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:24.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:24.689 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:25.691 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:25.691 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:25.691 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:46:25.692 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:46:25.692 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:25.692 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:26.115 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:26.707 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:46:26.703+0000 7faf5bac37c0 -1 osd.2 0 log_to_monitors true 2024-09-15T20:46:27.117 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:27.117 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:27.117 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:46:27.118 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:46:27.118 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:27.118 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:27.691 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:28.692 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:28.693 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:28.693 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:46:28.693 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:46:28.693 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:28.694 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:29.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:30.160 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:30.160 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:30.160 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:46:30.161 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:46:30.161 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:30.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:30.577 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:31.578 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:31.578 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:46:31.579 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:31.579 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:46:31.579 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:31.579 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:32.045 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:33.047 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:33.047 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:33.047 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:46:33.048 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:46:33.048 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:33.048 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:33.514 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:34.515 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:34.516 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:46:34.516 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:34.516 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:46:34.516 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:34.516 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:35.151 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:36.153 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:36.153 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:36.153 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:46:36.154 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:46:36.154 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:36.154 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:36.579 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:37.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:37.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:37.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:46:37.581 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:46:37.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:37.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:38.075 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:39.077 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:39.077 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:39.077 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:46:39.078 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:46:39.078 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:39.078 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:39.526 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:40.528 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:40.528 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:40.529 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:46:40.529 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:46:40.529 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:40.529 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:41.077 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:42.079 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:42.080 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:42.080 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:46:42.080 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:46:42.080 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:42.081 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:42.669 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:43.671 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:43.671 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:43.671 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:46:43.671 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:46:43.671 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:43.672 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:44.643 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:45.644 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:45.644 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:45.645 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:46:45.645 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:46:45.645 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:45.645 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:46.803 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:47.805 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:47.805 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:47.805 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:46:47.805 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:46:47.806 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:47.806 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:48.721 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:49.723 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:49.723 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:49.723 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:46:49.724 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:46:49.724 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:49.724 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:51.007 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:46:52.009 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:46:52.009 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:46:52.009 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:46:52.009 INFO:tasks.workunit.client.0.smithi139.stdout:18 2024-09-15T20:46:52.010 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:46:52.010 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:46:52.459 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1142441389,v1:127.0.0.1:6819/1142441389] [v2:127.0.0.1:6820/1142441389,v1:127.0.0.1:6821/1142441389] exists,up 37bcd18d-fac2-4f1c-92a0-af6c822e2dd4 2024-09-15T20:46:52.463 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:46:52.463 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:46:52.463 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:46:52.463 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2024-09-15T20:46:52.463 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-15T20:46:52.463 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:46:52.463 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T20:46:52.463 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:46:52.463 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/3 2024-09-15T20:46:52.463 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-15T20:46:52.463 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:46:52.463 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:46:52.464 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:46:52.464 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-09-15T20:46:52.464 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-09-15T20:46:52.464 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:46:52.464 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:46:52.464 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-15T20:46:52.464 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:46:52.464 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:46:52.464 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:46:52.464 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:46:52.464 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:46:52.464 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.84938 2024-09-15T20:46:52.464 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.84938/$cluster-$name.asok' 2024-09-15T20:46:52.464 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' 2024-09-15T20:46:52.465 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:46:52.465 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:46:52.465 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:46:52.465 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-15T20:46:52.465 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-15T20:46:52.465 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:46:52.465 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:46:52.465 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:46:52.465 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:46:52.465 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:46:52.465 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:46:52.465 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/3 2024-09-15T20:46:52.466 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:46:52.466 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e4b4b0b9-944b-4164-8e76-12d7b19380ff 2024-09-15T20:46:52.466 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 e4b4b0b9-944b-4164-8e76-12d7b19380ff' 2024-09-15T20:46:52.466 INFO:tasks.workunit.client.0.smithi139.stdout:add osd3 e4b4b0b9-944b-4164-8e76-12d7b19380ff 2024-09-15T20:46:52.467 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:46:52.481 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC8R+dmI2/OHBAAIRmBaX+UUeMNv4Mz+EuDFg== 2024-09-15T20:46:52.481 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC8R+dmI2/OHBAAIRmBaX+UUeMNv4Mz+EuDFg=="}' 2024-09-15T20:46:52.481 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e4b4b0b9-944b-4164-8e76-12d7b19380ff -i td/ec-error-rollforward/3/new.json 2024-09-15T20:46:53.191 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:46:53.211 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/3/new.json 2024-09-15T20:46:53.211 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC8R+dmI2/OHBAAIRmBaX+UUeMNv4Mz+EuDFg== --osd-uuid e4b4b0b9-944b-4164-8e76-12d7b19380ff 2024-09-15T20:46:53.238 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:46:53.234+0000 7fd3a5ca07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:46:53.247 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:46:53.246+0000 7fd3a5ca07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:46:53.248 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:46:53.246+0000 7fd3a5ca07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:46:53.249 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:46:53.246+0000 7fd3a5ca07c0 -1 bdev(0x55bf16b11c00 td/ec-error-rollforward/3/block) open stat got: (1) Operation not permitted 2024-09-15T20:46:53.249 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:46:53.246+0000 7fd3a5ca07c0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2024-09-15T20:47:00.105 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2024-09-15T20:47:00.105 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:47:00.106 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd3 key to auth repository 2024-09-15T20:47:00.106 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T20:47:00.106 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:47:00.668 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.3 2024-09-15T20:47:00.668 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T20:47:00.669 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:47:00.669 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:47:00.669 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:47:00.669 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:47:00.697 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:00.694+0000 7f26daccc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:00.729 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:00.726+0000 7f26daccc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:00.730 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:00.726+0000 7f26daccc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:01.149 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T20:47:01.149 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:47:01.149 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:47:01.149 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:47:01.150 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:47:01.150 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:47:01.150 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:01.150 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:47:01.150 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:01.151 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:01.587 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:02.218 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:02.214+0000 7f26daccc7c0 -1 Falling back to public interface 2024-09-15T20:47:02.589 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:02.590 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:02.590 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:47:02.590 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:47:02.590 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:02.590 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:03.034 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:04.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:04.036 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:04.036 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:47:04.036 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:47:04.036 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:04.036 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:04.484 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:05.121 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:05.118+0000 7f26daccc7c0 -1 osd.3 0 log_to_monitors true 2024-09-15T20:47:05.484 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:05.485 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:05.485 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:47:05.486 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:47:05.486 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:05.486 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:05.973 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:06.974 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:06.975 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:06.975 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:47:06.976 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:47:06.976 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:06.976 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:07.422 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:08.424 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:08.424 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:08.424 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:47:08.425 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:47:08.425 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:08.425 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:08.943 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:09.945 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:09.945 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:09.945 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:47:09.945 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:47:09.945 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:09.946 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:10.364 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:11.366 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:11.366 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:11.366 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:47:11.366 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:47:11.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:11.367 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:11.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:12.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:12.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:12.787 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:47:12.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:47:12.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:12.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:13.322 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:14.324 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:14.324 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:14.325 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:47:14.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:47:14.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:14.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:14.764 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:15.765 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:15.765 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:15.766 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:47:15.766 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:47:15.766 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:15.766 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:16.293 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:17.294 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:17.294 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:17.295 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:47:17.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:47:17.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:17.296 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:17.725 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:18.726 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:18.727 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:18.727 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:47:18.727 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:47:18.728 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:18.728 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:19.575 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:20.577 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:20.577 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:20.577 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:47:20.577 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:47:20.578 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:20.578 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:21.598 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:22.600 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:22.600 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:22.600 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:47:22.601 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:47:22.601 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:22.601 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:23.477 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:24.479 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:24.479 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:24.479 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:47:24.480 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:47:24.480 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:24.480 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:24.997 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:25.999 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:25.999 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:25.999 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:47:26.000 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:47:26.000 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:26.001 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:26.822 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:27.824 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:27.824 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:27.824 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:47:27.825 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:47:27.825 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:27.825 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:28.556 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:29.558 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:29.558 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:29.559 INFO:tasks.workunit.client.0.smithi139.stdout:18 2024-09-15T20:47:29.559 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:47:29.560 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:29.560 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:30.503 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:47:31.505 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:47:31.505 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:31.505 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:47:31.505 INFO:tasks.workunit.client.0.smithi139.stdout:19 2024-09-15T20:47:31.506 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:31.506 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:31.933 INFO:tasks.workunit.client.0.smithi139.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/342400769,v1:127.0.0.1:6827/342400769] [v2:127.0.0.1:6828/342400769,v1:127.0.0.1:6829/342400769] exists,up e4b4b0b9-944b-4164-8e76-12d7b19380ff 2024-09-15T20:47:31.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:47:31.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:47:31.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:47:31.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-09-15T20:47:32.553 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2024-09-15T20:47:33.460 INFO:tasks.workunit.client.0.smithi139.stderr:pool 'ec' already exists 2024-09-15T20:47:33.478 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2024-09-15T20:47:35.255 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2024-09-15T20:47:35.256 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 89592 2024-09-15T20:47:35.256 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=92396 2024-09-15T20:47:35.257 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2024-09-15T20:47:35.257 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2024-09-15T20:47:36.260 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 92417' 2024-09-15T20:47:36.260 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2024-09-15T20:47:36.260 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 92418' 2024-09-15T20:47:36.260 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2024-09-15T20:47:36.261 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 92419' 2024-09-15T20:47:36.261 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2024-09-15T20:47:36.261 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2024-09-15T20:47:37.261 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2024-09-15T20:47:37.262 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:47:37.263 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:47:37.263 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:47:37.263 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:47:37.263 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:47:37.577 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:47:37.577 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 92396 92417 92418 92419 2024-09-15T20:47:37.577 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2024-09-15T20:47:37.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2024-09-15T20:47:37.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-15T20:47:37.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:47:37.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:47:37.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:47:37.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/0 2024-09-15T20:47:37.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-15T20:47:37.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:47:37.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:47:37.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:47:37.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-09-15T20:47:37.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-09-15T20:47:37.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:47:37.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:47:37.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-15T20:47:37.582 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:47:37.582 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:47:37.582 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:47:37.583 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:47:37.583 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:47:37.583 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.84938 2024-09-15T20:47:37.583 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.84938/$cluster-$name.asok' 2024-09-15T20:47:37.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' 2024-09-15T20:47:37.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:47:37.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-15T20:47:37.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-15T20:47:37.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:47:37.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:47:37.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:47:37.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:47:37.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:47:37.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:47:37.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/0 2024-09-15T20:47:37.584 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T20:47:37.585 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:47:37.585 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:47:37.585 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/0/whoami 2024-09-15T20:47:37.586 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:47:37.586 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:47:37.586 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:47:37.586 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:47:37.612 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:37.613+0000 7f8682b6f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:37.615 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:37.613+0000 7f8682b6f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:37.617 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:37.617+0000 7f8682b6f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:38.019 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:47:38.019 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:47:38.019 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:47:38.019 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:47:38.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:47:38.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:38.020 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:47:38.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:47:38.021 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:47:38.021 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:38.463 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3721910055,v1:127.0.0.1:6803/3721910055] [v2:127.0.0.1:6804/3721910055,v1:127.0.0.1:6805/3721910055] exists,up 46ae8d2d-04be-48da-a686-147d13c7798e 2024-09-15T20:47:38.465 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:47:38.466 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:47:38.466 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:47:38.466 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2024-09-15T20:47:38.466 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-15T20:47:38.466 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:47:38.466 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:47:38.466 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:47:38.466 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/1 2024-09-15T20:47:38.466 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-15T20:47:38.466 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:47:38.466 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:47:38.466 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:47:38.466 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-09-15T20:47:38.467 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-09-15T20:47:38.467 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:47:38.467 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:47:38.467 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-15T20:47:38.467 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:47:38.467 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:47:38.467 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:47:38.467 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:47:38.467 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:47:38.467 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.84938 2024-09-15T20:47:38.467 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.84938/$cluster-$name.asok' 2024-09-15T20:47:38.468 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' 2024-09-15T20:47:38.468 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:47:38.468 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-15T20:47:38.468 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-15T20:47:38.468 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:47:38.468 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:47:38.469 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:47:38.469 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:47:38.469 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:47:38.469 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:47:38.469 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/1 2024-09-15T20:47:38.469 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.1 2024-09-15T20:47:38.470 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:47:38.470 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:47:38.470 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/1/whoami 2024-09-15T20:47:38.470 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:47:38.470 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:47:38.470 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:47:38.470 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:47:38.492 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:38.493+0000 7f42ee4817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:38.525 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:38.525+0000 7f42ee4817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:38.527 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:38.525+0000 7f42ee4817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:38.892 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:47:38.893 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:47:38.893 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:47:38.893 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:47:38.893 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:47:38.893 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:47:38.893 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:38.894 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:47:38.894 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:38.894 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:47:38.917 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:38.917+0000 7f8682b6f7c0 -1 Falling back to public interface 2024-09-15T20:47:39.345 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2546374193,v1:127.0.0.1:6811/2546374193] [v2:127.0.0.1:6812/2546374193,v1:127.0.0.1:6813/2546374193] exists,up 375b6031-d027-47c1-bddf-0d15d664926c 2024-09-15T20:47:39.350 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:47:39.350 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:47:39.350 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:47:39.350 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2024-09-15T20:47:39.350 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-15T20:47:39.351 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:47:39.351 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T20:47:39.351 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:47:39.351 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/2 2024-09-15T20:47:39.351 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-15T20:47:39.351 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:47:39.351 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:47:39.351 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:47:39.352 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-09-15T20:47:39.352 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-09-15T20:47:39.352 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:47:39.352 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:47:39.352 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-15T20:47:39.352 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:47:39.352 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:47:39.352 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:47:39.353 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:47:39.353 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:47:39.353 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.84938 2024-09-15T20:47:39.353 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.84938/$cluster-$name.asok' 2024-09-15T20:47:39.353 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' 2024-09-15T20:47:39.353 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:47:39.353 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-15T20:47:39.353 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-15T20:47:39.354 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:47:39.354 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:47:39.354 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:47:39.354 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:47:39.354 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:47:39.354 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:47:39.354 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/2 2024-09-15T20:47:39.355 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.2 2024-09-15T20:47:39.356 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T20:47:39.356 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:47:39.356 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/2/whoami 2024-09-15T20:47:39.356 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T20:47:39.356 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:47:39.356 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:47:39.357 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:47:39.383 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:39.381+0000 7fbc9dccb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:39.386 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:39.385+0000 7fbc9dccb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:39.387 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:39.385+0000 7fbc9dccb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:39.622 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:39.621+0000 7f42ee4817c0 -1 Falling back to public interface 2024-09-15T20:47:39.758 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T20:47:39.758 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:47:39.758 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:47:39.758 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:47:39.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:47:39.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:39.759 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:47:39.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:47:39.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:39.759 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:47:40.207 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1142441389,v1:127.0.0.1:6819/1142441389] [v2:127.0.0.1:6820/1142441389,v1:127.0.0.1:6821/1142441389] exists,up 37bcd18d-fac2-4f1c-92a0-af6c822e2dd4 2024-09-15T20:47:40.210 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:47:40.210 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:47:40.210 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:47:40.210 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2024-09-15T20:47:40.210 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-15T20:47:40.210 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:47:40.210 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T20:47:40.210 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:47:40.210 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/3 2024-09-15T20:47:40.211 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-15T20:47:40.211 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:47:40.211 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:47:40.211 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:47:40.211 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-09-15T20:47:40.211 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-09-15T20:47:40.211 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:47:40.211 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:47:40.211 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-15T20:47:40.211 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:47:40.211 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:47:40.211 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:47:40.211 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:47:40.212 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:47:40.212 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.84938 2024-09-15T20:47:40.212 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.84938/$cluster-$name.asok' 2024-09-15T20:47:40.212 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' 2024-09-15T20:47:40.212 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:47:40.212 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-15T20:47:40.212 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-15T20:47:40.212 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:47:40.212 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:47:40.212 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:47:40.212 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:47:40.212 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:47:40.212 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:47:40.212 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/3 2024-09-15T20:47:40.213 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.3 2024-09-15T20:47:40.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T20:47:40.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=cf99130b-034e-4a88-8943-c44307c8e8a5 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84938/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:47:40.214 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/3/whoami 2024-09-15T20:47:40.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T20:47:40.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:47:40.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:47:40.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:47:40.234 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:40.233+0000 7f38bc0c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:40.264 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:40.265+0000 7f38bc0c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:40.266 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:40.265+0000 7f38bc0c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:40.652 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T20:47:40.652 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:47:40.652 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:47:40.652 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:47:40.652 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:47:40.652 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:47:40.653 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:47:40.653 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:47:40.653 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:47:40.653 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:47:40.957 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:40.957+0000 7fbc9dccb7c0 -1 Falling back to public interface 2024-09-15T20:47:41.106 INFO:tasks.workunit.client.0.smithi139.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/342400769,v1:127.0.0.1:6827/342400769] [v2:127.0.0.1:6828/342400769,v1:127.0.0.1:6829/342400769] exists,up e4b4b0b9-944b-4164-8e76-12d7b19380ff 2024-09-15T20:47:41.107 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:47:41.107 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:47:41.107 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:47:41.107 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2024-09-15T20:47:41.107 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:47:41.107 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:47:41.107 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:47:41.107 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:47:41.108 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:47:41.108 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:47:41.108 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:47:41.108 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:47:41.108 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:47:41.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:47:41.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:47:41.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:47:41.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:47:41.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:47:41.199 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:47:41.509 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:41.509+0000 7f38bc0c37c0 -1 Falling back to public interface 2024-09-15T20:47:41.627 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:47:41.627 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:47:41.628 INFO:tasks.workunit.client.0.smithi139.stderr:2 2024-09-15T20:47:41.628 INFO:tasks.workunit.client.0.smithi139.stderr:3' 2024-09-15T20:47:41.628 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:47:41.628 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:47:41.628 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:47:43.015 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:43.013+0000 7f8682b6f7c0 -1 osd.0 26 log_to_monitors true 2024-09-15T20:47:43.720 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:43.717+0000 7f42ee4817c0 -1 osd.1 26 log_to_monitors true 2024-09-15T20:47:45.654 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:45.653+0000 7fbc9dccb7c0 -1 osd.2 25 log_to_monitors true 2024-09-15T20:47:45.770 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:45.769+0000 7f38bc0c37c0 -1 osd.3 26 log_to_monitors true 2024-09-15T20:47:46.980 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051587 2024-09-15T20:47:46.981 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051587 2024-09-15T20:47:46.981 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051587' 2024-09-15T20:47:46.981 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:47:46.981 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:47:47.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953475 2024-09-15T20:47:47.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953475 2024-09-15T20:47:47.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051587 1-137438953475' 2024-09-15T20:47:47.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:47:47.140 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:47:47.329 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822659 2024-09-15T20:47:47.329 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822659 2024-09-15T20:47:47.330 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051587 1-137438953475 2-154618822659' 2024-09-15T20:47:47.330 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:47:47.330 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:47:47.486 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822659 2024-09-15T20:47:47.486 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822659 2024-09-15T20:47:47.486 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051587 1-137438953475 2-154618822659 3-154618822659' 2024-09-15T20:47:47.486 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:47:47.487 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-124554051587 2024-09-15T20:47:47.487 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:47:47.488 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:47:47.488 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-124554051587 2024-09-15T20:47:47.489 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:47:47.489 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051587 2024-09-15T20:47:47.489 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 124554051587' 2024-09-15T20:47:47.490 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 124554051587 2024-09-15T20:47:47.490 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:47:47.970 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 124554051587 2024-09-15T20:47:47.970 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:47:48.971 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:47:48.972 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:47:49.398 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051588 -lt 124554051587 2024-09-15T20:47:49.398 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:47:49.399 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:47:49.399 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-137438953475 2024-09-15T20:47:49.400 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:47:49.401 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-137438953475 2024-09-15T20:47:49.401 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:47:49.402 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953475 2024-09-15T20:47:49.402 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.1 seq 137438953475 2024-09-15T20:47:49.402 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 137438953475' 2024-09-15T20:47:49.402 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:47:49.880 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953475 -lt 137438953475 2024-09-15T20:47:49.880 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:47:49.881 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-154618822659 2024-09-15T20:47:49.882 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:47:49.883 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:47:49.884 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-154618822659 2024-09-15T20:47:49.885 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:47:49.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822659 2024-09-15T20:47:49.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2024-09-15T20:47:49.886 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 154618822659 2024-09-15T20:47:49.886 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:47:50.345 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822659 -lt 154618822659 2024-09-15T20:47:50.346 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:47:50.346 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:47:50.346 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-154618822659 2024-09-15T20:47:50.347 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:47:50.348 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-154618822659 2024-09-15T20:47:50.348 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:47:50.349 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822659 2024-09-15T20:47:50.349 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 154618822659' 2024-09-15T20:47:50.350 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.3 seq 154618822659 2024-09-15T20:47:50.350 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:47:50.771 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822659 -lt 154618822659 2024-09-15T20:47:50.771 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:47:50.771 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:47:50.772 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:47:51.376 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-15T20:47:51.376 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:47:51.377 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:47:51.377 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:47:51.377 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:47:51.377 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:47:51.377 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:47:51.378 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:47:51.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-15T20:47:51.784 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:47:51.785 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:47:51.785 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:47:52.346 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-15T20:47:52.347 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:47:52.347 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:47:52.347 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2024-09-15T20:47:52.347 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ec-error-rollforward 2024-09-15T20:47:52.347 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:47:52.347 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-15T20:47:52.348 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:47:52.349 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:47:52.349 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:47:52.349 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:47:52.349 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:47:52.458 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:47:52.459 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:47:52.459 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:47:52.459 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:47:52.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:47:52.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:47:52.461 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:47:52.462 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:47:52.462 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:47:52.462 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:47:52.462 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:47:52.463 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:47:52.464 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:47:52.464 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ec-error-rollforward 2024-09-15T20:47:52.483 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:47:52.483 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:47:52.483 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.84938 2024-09-15T20:47:52.483 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.84938 2024-09-15T20:47:52.485 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:47:52.485 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:47:52.485 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-15T20:47:52.485 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/ec-error-rollforward 0 2024-09-15T20:47:52.485 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ec-error-rollforward 2024-09-15T20:47:52.485 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-15T20:47:52.485 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-15T20:47:52.485 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:47:52.485 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:47:52.485 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:47:52.485 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:47:52.485 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:47:52.486 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:47:52.487 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:47:52.488 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:47:52.488 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:47:52.489 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:47:52.489 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:47:52.489 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:47:52.490 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:47:52.490 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:47:52.490 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:47:52.490 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:47:52.491 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:47:52.492 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T20:47:52.492 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ec-error-rollforward 2024-09-15T20:47:52.493 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:47:52.493 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:47:52.493 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.84938 2024-09-15T20:47:52.493 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.84938 2024-09-15T20:47:52.494 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:47:52.495 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:47:52.495 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-15T20:47:52.495 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T20:47:52.495 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T20:47:52.548 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2024-09-15T20:47:52.548 DEBUG:teuthology.orchestra.run.smithi139:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2024-09-15T20:47:52.601 INFO:tasks.workunit.client.0.smithi139.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T20:47:52.605 INFO:tasks.workunit.client.0.smithi139.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T20:47:52.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-15T20:47:52.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-15T20:47:52.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-15T20:47:52.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-15T20:47:52.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-15T20:47:52.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-15T20:47:52.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-15T20:47:52.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-15T20:47:52.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-bench 2024-09-15T20:47:52.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2024-09-15T20:47:52.607 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2024-09-15T20:47:52.607 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2024-09-15T20:47:52.607 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2024-09-15T20:47:52.607 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2024-09-15T20:47:52.607 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2024-09-15T20:47:52.609 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=e2ca1c27-32cd-4e06-a7b5-17a405490109 --auth-supported=none ' 2024-09-15T20:47:52.609 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2024-09-15T20:47:52.609 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-09-15T20:47:52.609 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2024-09-15T20:47:52.609 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T20:47:52.609 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2024-09-15T20:47:52.609 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2024-09-15T20:47:52.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2024-09-15T20:47:52.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bench 2024-09-15T20:47:52.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bench 2024-09-15T20:47:52.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bench 2024-09-15T20:47:52.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:47:52.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bench KILL 2024-09-15T20:47:52.610 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:47:52.610 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:47:52.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:47:52.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:47:52.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:47:52.611 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:47:52.611 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:47:52.611 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:47:52.612 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:47:52.612 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:47:52.612 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:47:52.613 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:47:52.613 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:47:52.613 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:47:52.614 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:47:52.614 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:47:52.615 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:47:52.616 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:47:52.616 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bench 2024-09-15T20:47:52.617 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:47:52.617 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:47:52.617 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.94959 2024-09-15T20:47:52.617 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.94959 2024-09-15T20:47:52.618 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:47:52.618 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:47:52.618 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bench 2024-09-15T20:47:52.619 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:47:52.619 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:47:52.619 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.94959 2024-09-15T20:47:52.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.94959 2024-09-15T20:47:52.621 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:47:52.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-15T20:47:52.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-15T20:47:52.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:47:52.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2024-09-15T20:47:52.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2024-09-15T20:47:52.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2024-09-15T20:47:52.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2024-09-15T20:47:52.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bench 2024-09-15T20:47:52.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T20:47:52.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T20:47:52.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T20:47:52.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bench/a 2024-09-15T20:47:52.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2024-09-15T20:47:53.091 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T20:47:53.091 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:47:53.091 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:47:53.092 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:47:53.092 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:47:53.092 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.94959 2024-09-15T20:47:53.092 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.94959/$cluster-$name.asok' 2024-09-15T20:47:53.093 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.94959/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T20:47:53.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T20:47:53.304 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T20:47:53.304 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:47:53.304 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:47:53.304 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T20:47:53.305 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:47:53.305 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T20:47:53.305 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:47:53.305 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:47:53.306 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:47:53.306 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:47:53.306 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.94959 2024-09-15T20:47:53.306 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.94959/ceph-mon.a.asok 2024-09-15T20:47:53.306 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:47:53.306 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.94959/ceph-mon.a.asok config get fsid 2024-09-15T20:47:53.416 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T20:47:53.416 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:47:53.417 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:47:53.417 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T20:47:53.417 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:47:53.418 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T20:47:53.418 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:47:53.418 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:47:53.418 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:47:53.418 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:47:53.418 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.94959 2024-09-15T20:47:53.418 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.94959/ceph-mon.a.asok 2024-09-15T20:47:53.419 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:47:53.419 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.94959/ceph-mon.a.asok config get mon_host 2024-09-15T20:47:53.520 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2024-09-15T20:47:53.520 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bench 2024-09-15T20:47:53.520 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T20:47:53.520 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T20:47:53.520 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T20:47:53.520 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bench/x 2024-09-15T20:47:53.520 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T20:47:54.433 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T20:47:54.434 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:47:54.434 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:47:54.434 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:47:54.434 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:47:54.435 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.94959 2024-09-15T20:47:54.435 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.94959/$cluster-$name.asok' 2024-09-15T20:47:54.435 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:47:54.436 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.94959/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:47:54.470 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2024-09-15T20:47:54.470 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bench 2024-09-15T20:47:54.470 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:47:54.470 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T20:47:54.470 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:47:54.471 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bench/0 2024-09-15T20:47:54.471 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e2ca1c27-32cd-4e06-a7b5-17a405490109 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2024-09-15T20:47:54.471 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:47:54.471 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:47:54.471 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:47:54.471 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2024-09-15T20:47:54.471 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2024-09-15T20:47:54.471 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:47:54.471 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:47:54.472 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2024-09-15T20:47:54.472 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:47:54.472 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:47:54.472 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:47:54.472 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:47:54.472 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:47:54.472 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.94959 2024-09-15T20:47:54.472 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.94959/$cluster-$name.asok' 2024-09-15T20:47:54.472 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.94959/$cluster-$name.asok' 2024-09-15T20:47:54.473 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:47:54.473 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:47:54.473 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:47:54.473 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2024-09-15T20:47:54.473 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2024-09-15T20:47:54.473 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:47:54.473 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:47:54.473 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:47:54.473 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:47:54.473 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:47:54.473 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:47:54.473 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bench/0 2024-09-15T20:47:54.473 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:47:54.474 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5a67da6a-dc30-4709-a700-94e725355ce3 2024-09-15T20:47:54.474 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5a67da6a-dc30-4709-a700-94e725355ce3' 2024-09-15T20:47:54.474 INFO:tasks.workunit.client.0.smithi139.stdout:add osd0 5a67da6a-dc30-4709-a700-94e725355ce3 2024-09-15T20:47:54.474 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:47:54.485 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD6R+dmqAIEHRAAqxhxz9JCViqjkLowpgoS7w== 2024-09-15T20:47:54.485 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD6R+dmqAIEHRAAqxhxz9JCViqjkLowpgoS7w=="}' 2024-09-15T20:47:54.485 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5a67da6a-dc30-4709-a700-94e725355ce3 -i td/osd-bench/0/new.json 2024-09-15T20:47:54.789 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:47:54.804 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bench/0/new.json 2024-09-15T20:47:54.806 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e2ca1c27-32cd-4e06-a7b5-17a405490109 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.94959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD6R+dmqAIEHRAAqxhxz9JCViqjkLowpgoS7w== --osd-uuid 5a67da6a-dc30-4709-a700-94e725355ce3 2024-09-15T20:47:54.842 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:54.841+0000 7f41777d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:54.866 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:54.865+0000 7f41777d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:54.867 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:54.865+0000 7f41777d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:47:54.867 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:54.865+0000 7f41777d07c0 -1 bdev(0x55986f65a800 td/osd-bench/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:47:54.867 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:47:54.869+0000 7f41777d07c0 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2024-09-15T20:48:01.268 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bench/0/keyring 2024-09-15T20:48:01.268 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:48:01.270 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd0 key to auth repository 2024-09-15T20:48:01.271 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T20:48:01.271 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:48:01.840 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T20:48:01.841 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T20:48:01.841 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:48:01.841 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e2ca1c27-32cd-4e06-a7b5-17a405490109 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.94959/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:48:01.841 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:48:01.841 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:48:01.870 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:48:01.869+0000 7f02d89267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:48:01.899 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:48:01.897+0000 7f02d89267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:48:01.900 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:48:01.901+0000 7f02d89267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:48:02.300 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T20:48:02.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:48:02.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:48:02.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:48:02.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:48:02.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:02.301 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:48:02.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:48:02.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:02.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:48:02.743 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:03.671 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:48:03.669+0000 7f02d89267c0 -1 Falling back to public interface 2024-09-15T20:48:03.745 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:03.745 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:03.745 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:48:03.746 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:48:03.746 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:03.746 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:48:04.158 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:05.159 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:05.160 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:05.160 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:48:05.161 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:48:05.161 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:05.161 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:48:05.639 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:06.644 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:06.644 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:06.645 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:48:06.645 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:48:06.645 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:06.645 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:48:06.795 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:48:06.792+0000 7f02d89267c0 -1 osd.0 0 log_to_monitors true 2024-09-15T20:48:07.114 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:08.115 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:08.115 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:08.116 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:48:08.116 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:48:08.116 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:08.116 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:48:08.656 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:09.658 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:09.658 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:09.658 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:48:09.659 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:48:09.659 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:09.659 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:48:10.076 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:11.077 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:11.078 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:11.078 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:48:11.078 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:48:11.078 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:11.079 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:48:11.680 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:12.682 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:12.682 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:12.682 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:48:12.683 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:48:12.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:12.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:48:13.146 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:14.147 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:14.147 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:14.148 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:48:14.148 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:48:14.148 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:14.148 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:48:14.690 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:15.691 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:15.692 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:15.692 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:48:15.692 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:48:15.692 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:15.693 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:48:16.218 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:17.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:17.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:17.220 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:48:17.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:48:17.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:17.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:48:17.805 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:18.807 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:18.807 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:18.807 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:48:18.807 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:48:18.808 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:18.808 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:48:19.379 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:20.380 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:20.380 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:20.381 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:48:20.381 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:48:20.381 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:20.381 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:48:21.354 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:22.358 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:22.358 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:22.358 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:48:22.358 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:48:22.359 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:22.359 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:48:23.411 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:24.412 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:24.413 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:48:24.413 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:24.413 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:48:24.413 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:24.414 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:48:25.308 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:26.310 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:26.310 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:48:26.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:26.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:48:26.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:26.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:48:27.300 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:28.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:28.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:28.302 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:48:28.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:48:28.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:28.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:48:29.091 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:30.093 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:48:30.093 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:30.093 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:30.093 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:48:30.094 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:30.094 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:48:31.104 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:48:32.105 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:48:32.105 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:48:32.106 INFO:tasks.workunit.client.0.smithi139.stdout:18 2024-09-15T20:48:32.106 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:48:32.106 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:48:32.106 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:48:32.603 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/73467688,v1:127.0.0.1:6803/73467688] [v2:127.0.0.1:6804/73467688,v1:127.0.0.1:6805/73467688] exists,up 5a67da6a-dc30-4709-a700-94e725355ce3 2024-09-15T20:48:32.603 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:48:32.603 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:48:32.603 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:48:32.603 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2024-09-15T20:48:32.603 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2024-09-15T20:48:32.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2024-09-15T20:48:32.621 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2024-09-15T20:48:32.621 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2024-09-15T20:48:32.633 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2024-09-15T20:48:32.634 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2024-09-15T20:48:32.634 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2024-09-15T20:48:32.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2024-09-15T20:48:32.652 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2024-09-15T20:48:32.652 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2024-09-15T20:48:32.668 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2024-09-15T20:48:32.668 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2024-09-15T20:48:32.668 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/osd-bench 2024-09-15T20:48:32.668 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-15T20:48:32.668 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=osd_bench_max_block_size 2024-09-15T20:48:32.668 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-15T20:48:32.668 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-15T20:48:32.668 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: ceph tell osd.0 bench 1024 67108865 2024-09-15T20:48:32.798 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-15T20:48:32.798 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-15T20:48:32.798 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2024-09-15T20:48:32.799 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-15T20:48:32.799 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2024-09-15T20:48:32.800 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2024-09-15T20:48:32.800 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2024-09-15T20:48:32.800 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/osd-bench 2024-09-15T20:48:32.800 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-15T20:48:32.800 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=bench_small_size_max_iops 2024-09-15T20:48:32.800 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-15T20:48:32.800 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-15T20:48:32.800 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: ceph tell osd.0 bench 3072001 1024 2024-09-15T20:48:32.947 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-15T20:48:32.948 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-15T20:48:32.948 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2024-09-15T20:48:32.949 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-15T20:48:32.949 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2024-09-15T20:48:32.949 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2024-09-15T20:48:32.950 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2024-09-15T20:48:32.950 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/osd-bench 2024-09-15T20:48:32.950 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-15T20:48:32.950 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=osd_bench_large_size_max_throughput 2024-09-15T20:48:32.950 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-15T20:48:32.950 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-15T20:48:32.950 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2024-09-15T20:48:33.091 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-15T20:48:33.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-15T20:48:33.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2024-09-15T20:48:33.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-15T20:48:33.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2024-09-15T20:49:29.263 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:49:29.264 INFO:tasks.workunit.client.0.smithi139.stdout: "bytes_written": 1073741824, 2024-09-15T20:49:29.264 INFO:tasks.workunit.client.0.smithi139.stdout: "blocksize": 4194304, 2024-09-15T20:49:29.264 INFO:tasks.workunit.client.0.smithi139.stdout: "elapsed_sec": 26.912893922999999, 2024-09-15T20:49:29.264 INFO:tasks.workunit.client.0.smithi139.stdout: "bytes_per_sec": 39896929.221809573, 2024-09-15T20:49:29.264 INFO:tasks.workunit.client.0.smithi139.stdout: "iops": 9.5121691755794462 2024-09-15T20:49:29.264 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:49:29.291 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2024-09-15T20:49:29.799 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:49:29.799 INFO:tasks.workunit.client.0.smithi139.stdout: "bytes_written": 10, 2024-09-15T20:49:29.799 INFO:tasks.workunit.client.0.smithi139.stdout: "blocksize": 14456, 2024-09-15T20:49:29.799 INFO:tasks.workunit.client.0.smithi139.stdout: "elapsed_sec": 0.033483959000000001, 2024-09-15T20:49:29.799 INFO:tasks.workunit.client.0.smithi139.stdout: "bytes_per_sec": 298.65046722820318, 2024-09-15T20:49:29.799 INFO:tasks.workunit.client.0.smithi139.stdout: "iops": 0.020659274158010735 2024-09-15T20:49:29.799 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:49:29.821 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2024-09-15T20:49:30.099 INFO:tasks.workunit.client.0.smithi139.stdout:{ 2024-09-15T20:49:30.100 INFO:tasks.workunit.client.0.smithi139.stdout: "bytes_written": 1, 2024-09-15T20:49:30.100 INFO:tasks.workunit.client.0.smithi139.stdout: "blocksize": 14456, 2024-09-15T20:49:30.100 INFO:tasks.workunit.client.0.smithi139.stdout: "elapsed_sec": 0.032971192000000003, 2024-09-15T20:49:30.100 INFO:tasks.workunit.client.0.smithi139.stdout: "bytes_per_sec": 30.329507043603396, 2024-09-15T20:49:30.100 INFO:tasks.workunit.client.0.smithi139.stdout: "iops": 0.0020980566576925424 2024-09-15T20:49:30.100 INFO:tasks.workunit.client.0.smithi139.stdout:} 2024-09-15T20:49:30.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2024-09-15T20:49:30.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bench 2024-09-15T20:49:30.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:49:30.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bench KILL 2024-09-15T20:49:30.126 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:49:30.126 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:49:30.126 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:49:30.126 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:49:30.126 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:49:30.236 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:49:30.237 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:49:30.237 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:49:30.237 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:49:30.239 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:49:30.239 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:49:30.239 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:49:30.240 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:49:30.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:49:30.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:49:30.241 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:49:30.242 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:49:30.242 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:49:30.243 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bench 2024-09-15T20:49:30.272 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:49:30.272 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:49:30.272 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.94959 2024-09-15T20:49:30.272 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.94959 2024-09-15T20:49:30.274 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:49:30.274 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:49:30.274 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-15T20:49:30.274 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-bench 0 2024-09-15T20:49:30.274 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bench 2024-09-15T20:49:30.274 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-15T20:49:30.274 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bench KILL 2024-09-15T20:49:30.275 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:49:30.275 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:49:30.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:49:30.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:49:30.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:49:30.276 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:49:30.276 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:49:30.277 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:49:30.278 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:49:30.278 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:49:30.278 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:49:30.279 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:49:30.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:49:30.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:49:30.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:49:30.280 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:49:30.281 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:49:30.282 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T20:49:30.282 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bench 2024-09-15T20:49:30.283 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:49:30.284 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:49:30.284 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.94959 2024-09-15T20:49:30.284 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.94959 2024-09-15T20:49:30.285 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:49:30.285 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:49:30.285 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-15T20:49:30.286 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T20:49:30.286 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T20:49:30.338 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2024-09-15T20:49:30.338 DEBUG:teuthology.orchestra.run.smithi139:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2024-09-15T20:49:30.391 INFO:tasks.workunit.client.0.smithi139.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T20:49:30.398 INFO:tasks.workunit.client.0.smithi139.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T20:49:30.398 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-15T20:49:30.398 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-15T20:49:30.398 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-15T20:49:30.398 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-15T20:49:30.398 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-15T20:49:30.398 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-15T20:49:30.398 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-15T20:49:30.398 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-15T20:49:30.398 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-bluefs-volume-ops 2024-09-15T20:49:30.398 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:49:30.398 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2024-09-15T20:49:30.399 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: set 2024-09-15T20:49:30.399 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T20:49:30.402 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: local 'funcs=TEST_bluestore 2024-09-15T20:49:30.402 INFO:tasks.workunit.client.0.smithi139.stderr:TEST_bluestore2 2024-09-15T20:49:30.402 INFO:tasks.workunit.client.0.smithi139.stderr:TEST_bluestore_expand' 2024-09-15T20:49:30.402 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-15T20:49:30.402 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-15T20:49:30.402 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:49:30.402 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bluefs-volume-ops 2024-09-15T20:49:30.402 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:49:30.402 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T20:49:30.402 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-15T20:49:30.403 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T20:49:30.403 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T20:49:30.403 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T20:49:30.403 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T20:49:30.403 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T20:49:30.404 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T20:49:30.404 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T20:49:30.405 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T20:49:30.405 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T20:49:30.406 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T20:49:30.407 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T20:49:30.407 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T20:49:30.407 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:49:30.408 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T20:49:30.408 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T20:49:30.408 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:49:30.409 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:49:30.410 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T20:49:30.410 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-15T20:49:30.412 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T20:49:30.412 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:49:30.412 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:49:30.412 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.97169 2024-09-15T20:49:30.413 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T20:49:30.413 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T20:49:30.413 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-15T20:49:30.414 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T20:49:30.414 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:49:30.414 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:49:30.415 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.97169 2024-09-15T20:49:30.416 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T20:49:30.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-15T20:49:30.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-15T20:49:30.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T20:49:30.417 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-15T20:49:30.418 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore td/osd-bluefs-volume-ops 2024-09-15T20:49:30.418 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:49:30.418 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: ulimit -n 2024-09-15T20:49:30.418 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: local flimit=4096 2024-09-15T20:49:30.418 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: TEST_bluestore: '[' 4096 -lt 1536 ']' 2024-09-15T20:49:30.418 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: export CEPH_MON=127.0.0.1:7146 2024-09-15T20:49:30.418 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: CEPH_MON=127.0.0.1:7146 2024-09-15T20:49:30.418 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:27: TEST_bluestore: export CEPH_ARGS 2024-09-15T20:49:30.419 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: uuidgen 2024-09-15T20:49:30.419 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: CEPH_ARGS+='--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none ' 2024-09-15T20:49:30.420 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:29: TEST_bluestore: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-15T20:49:30.420 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2024-09-15T20:49:30.420 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:31: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-15T20:49:30.420 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-15T20:49:30.420 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2024-09-15T20:49:30.420 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:34: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2024-09-15T20:49:30.420 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:35: TEST_bluestore: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-15T20:49:30.420 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2024-09-15T20:49:30.420 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:49:30.420 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T20:49:30.420 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T20:49:30.420 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T20:49:30.420 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-15T20:49:30.420 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-15T20:49:30.918 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T20:49:30.919 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:49:30.919 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:49:30.919 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:49:30.919 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:49:30.919 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:49:30.919 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:49:30.919 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T20:49:31.131 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T20:49:31.132 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T20:49:31.132 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:49:31.132 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:49:31.132 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T20:49:31.133 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T20:49:31.133 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:49:31.133 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:49:31.133 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:49:31.134 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:49:31.134 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:49:31.134 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:49:31.134 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.97169/ceph-mon.a.asok 2024-09-15T20:49:31.134 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:49:31.134 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.97169/ceph-mon.a.asok config get fsid 2024-09-15T20:49:31.249 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T20:49:31.250 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T20:49:31.250 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T20:49:31.250 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T20:49:31.251 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T20:49:31.251 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T20:49:31.251 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T20:49:31.251 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T20:49:31.251 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T20:49:31.251 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:49:31.251 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:49:31.251 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.97169/ceph-mon.a.asok 2024-09-15T20:49:31.252 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T20:49:31.252 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.97169/ceph-mon.a.asok config get mon_host 2024-09-15T20:49:31.351 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2024-09-15T20:49:31.351 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:49:31.351 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T20:49:31.351 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T20:49:31.351 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T20:49:31.352 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-15T20:49:31.352 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T20:49:32.277 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T20:49:32.277 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:49:32.277 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:49:32.277 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:49:32.278 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:49:32.278 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:49:32.278 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:49:32.278 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:49:32.279 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T20:49:32.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2024-09-15T20:49:32.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:49:32.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:49:32.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T20:49:32.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:49:32.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-15T20:49:32.316 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T20:49:32.317 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:49:32.317 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:49:32.317 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:49:32.317 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-15T20:49:32.317 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-15T20:49:32.317 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:49:32.317 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:49:32.317 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T20:49:32.317 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:49:32.317 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:49:32.317 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:49:32.317 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:49:32.317 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:49:32.317 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:49:32.317 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:49:32.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:49:32.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:49:32.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:49:32.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:49:32.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T20:49:32.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T20:49:32.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:49:32.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:49:32.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:49:32.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:49:32.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:49:32.320 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:49:32.320 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-15T20:49:32.320 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:49:32.320 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3b45a55c-3ec6-4b29-933b-e92e4449d4c3 2024-09-15T20:49:32.320 INFO:tasks.workunit.client.0.smithi139.stdout:add osd0 3b45a55c-3ec6-4b29-933b-e92e4449d4c3 2024-09-15T20:49:32.320 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3b45a55c-3ec6-4b29-933b-e92e4449d4c3' 2024-09-15T20:49:32.320 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:49:32.336 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBcSOdmA9siFBAAa9j0VnzZAZHkoiiK6RqGdw== 2024-09-15T20:49:32.336 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBcSOdmA9siFBAAa9j0VnzZAZHkoiiK6RqGdw=="}' 2024-09-15T20:49:32.337 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3b45a55c-3ec6-4b29-933b-e92e4449d4c3 -i td/osd-bluefs-volume-ops/0/new.json 2024-09-15T20:49:32.649 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:49:32.665 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-15T20:49:32.667 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBcSOdmA9siFBAAa9j0VnzZAZHkoiiK6RqGdw== --osd-uuid 3b45a55c-3ec6-4b29-933b-e92e4449d4c3 2024-09-15T20:49:32.699 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:49:32.694+0000 7ff16575c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:49:32.727 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:49:32.722+0000 7ff16575c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:49:32.728 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:49:32.726+0000 7ff16575c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:49:32.728 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:49:32.726+0000 7ff16575c7c0 -1 bdev(0x5645e4dcfc00 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-09-15T20:49:32.729 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:49:32.726+0000 7ff16575c7c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-15T20:49:42.396 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-15T20:49:42.397 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:49:42.397 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd0 key to auth repository 2024-09-15T20:49:42.397 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T20:49:42.397 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:49:42.968 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T20:49:42.969 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T20:49:42.969 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:49:42.969 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:49:42.969 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:49:42.969 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:49:42.998 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:49:42.994+0000 7f76713bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:49:43.012 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:49:43.010+0000 7f76713bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:49:43.015 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:49:43.010+0000 7f76713bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:49:43.380 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T20:49:43.380 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:49:43.381 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:49:43.381 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:49:43.381 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:49:43.381 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:43.381 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:49:43.381 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:49:43.382 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:43.382 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:49:43.823 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:44.583 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:49:44.578+0000 7f76713bf7c0 -1 Falling back to public interface 2024-09-15T20:49:44.824 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:44.824 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:44.824 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:49:44.825 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:49:44.825 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:44.825 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:49:45.236 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:46.237 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:46.237 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:46.238 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:49:46.238 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:49:46.238 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:46.238 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:49:46.671 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:47.673 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:47.673 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:47.673 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:49:47.674 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:49:47.674 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:47.674 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:49:48.106 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:49.107 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:49.107 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:49.108 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:49:49.108 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:49:49.108 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:49.108 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:49:49.528 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:49.827 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:49:49.826+0000 7f76713bf7c0 -1 osd.0 0 log_to_monitors true 2024-09-15T20:49:50.530 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:50.530 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:50.530 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:49:50.530 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:49:50.531 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:50.531 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:49:51.145 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:52.146 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:52.147 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:52.147 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:49:52.147 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:49:52.147 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:52.147 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:49:52.607 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:53.609 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:53.609 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:53.609 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:49:53.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:49:53.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:53.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:49:54.147 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:55.148 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:55.148 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:55.149 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:49:55.149 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:49:55.149 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:55.149 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:49:55.578 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:56.579 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:56.580 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:56.580 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:49:56.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:49:56.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:56.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:49:57.199 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:58.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:58.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:58.201 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:49:58.202 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:49:58.202 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:58.202 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:49:58.629 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:49:59.631 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:49:59.631 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:49:59.631 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:49:59.632 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:49:59.632 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:49:59.632 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:50:00.094 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:01.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:01.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:01.096 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:50:01.097 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:50:01.097 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:01.097 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:50:01.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:02.620 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:02.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:02.621 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:50:02.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:50:02.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:02.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:50:03.119 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:04.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:04.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:04.122 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:50:04.123 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:50:04.123 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:04.123 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:50:04.961 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:05.964 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:05.964 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:05.964 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:50:05.965 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:50:05.965 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:05.965 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:50:06.562 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:07.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:07.564 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:50:07.565 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:07.565 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:50:07.565 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:07.565 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:50:08.710 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:09.711 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:09.712 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:09.712 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:50:09.712 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:50:09.712 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:09.713 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:50:10.961 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:11.962 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:11.962 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:11.962 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:50:11.963 INFO:tasks.workunit.client.0.smithi139.stdout:18 2024-09-15T20:50:11.963 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:11.963 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:50:12.811 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:13.813 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:13.813 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:13.813 INFO:tasks.workunit.client.0.smithi139.stdout:19 2024-09-15T20:50:13.813 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:50:13.814 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:13.814 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:50:14.674 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:15.675 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:15.675 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:15.676 INFO:tasks.workunit.client.0.smithi139.stdout:20 2024-09-15T20:50:15.676 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T20:50:15.676 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:15.676 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:50:16.218 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3676436028,v1:127.0.0.1:6803/3676436028] [v2:127.0.0.1:6804/3676436028,v1:127.0.0.1:6805/3676436028] exists,up 3b45a55c-3ec6-4b29-933b-e92e4449d4c3 2024-09-15T20:50:16.218 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:50:16.218 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:50:16.218 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:50:16.219 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-15T20:50:16.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=98635 2024-09-15T20:50:16.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2024-09-15T20:50:16.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:50:16.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:50:16.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-15T20:50:16.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:50:16.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-15T20:50:16.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T20:50:16.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:50:16.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:50:16.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:50:16.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-15T20:50:16.222 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-15T20:50:16.222 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:50:16.222 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:50:16.222 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T20:50:16.222 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:50:16.222 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:50:16.222 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:50:16.222 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:50:16.222 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:50:16.222 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:50:16.222 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:50:16.223 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:50:16.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:50:16.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:50:16.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:50:16.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T20:50:16.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T20:50:16.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:50:16.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:50:16.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:50:16.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:50:16.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:50:16.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:50:16.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-15T20:50:16.225 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:50:16.225 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a49f13f9-33e9-4f82-b5ec-e8b428f16326 2024-09-15T20:50:16.225 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 a49f13f9-33e9-4f82-b5ec-e8b428f16326' 2024-09-15T20:50:16.225 INFO:tasks.workunit.client.0.smithi139.stdout:add osd1 a49f13f9-33e9-4f82-b5ec-e8b428f16326 2024-09-15T20:50:16.226 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:50:16.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCISOdmeJl4DhAAQbQlPXcfP7k4mnhOX74WfQ== 2024-09-15T20:50:16.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCISOdmeJl4DhAAQbQlPXcfP7k4mnhOX74WfQ=="}' 2024-09-15T20:50:16.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a49f13f9-33e9-4f82-b5ec-e8b428f16326 -i td/osd-bluefs-volume-ops/1/new.json 2024-09-15T20:50:16.731 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:50:16.747 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2024-09-15T20:50:16.747 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCISOdmeJl4DhAAQbQlPXcfP7k4mnhOX74WfQ== --osd-uuid a49f13f9-33e9-4f82-b5ec-e8b428f16326 2024-09-15T20:50:16.779 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:50:16.777+0000 7fbe7e1337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:50:16.793 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:50:16.793+0000 7fbe7e1337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:50:16.794 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:50:16.793+0000 7fbe7e1337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:50:16.795 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:50:16.793+0000 7fbe7e1337c0 -1 bdev(0x556f8388dc00 td/osd-bluefs-volume-ops/1/block) open stat got: (1) Operation not permitted 2024-09-15T20:50:16.795 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:50:16.793+0000 7fbe7e1337c0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2024-09-15T20:50:27.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2024-09-15T20:50:27.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:50:27.097 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd1 key to auth repository 2024-09-15T20:50:27.097 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-15T20:50:27.097 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:50:27.669 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.1 2024-09-15T20:50:27.670 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-15T20:50:27.671 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:50:27.671 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:50:27.671 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:50:27.671 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:50:27.700 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:50:27.697+0000 7f0b87d947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:50:27.704 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:50:27.705+0000 7f0b87d947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:50:27.705 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:50:27.705+0000 7f0b87d947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:50:28.086 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-15T20:50:28.086 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:50:28.086 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:50:28.086 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:50:28.087 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:50:28.087 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:50:28.087 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:28.087 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:50:28.087 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:28.087 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:28.543 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:29.544 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:29.544 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:29.544 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:50:29.545 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:50:29.545 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:29.545 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:29.760 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:50:29.757+0000 7f0b87d947c0 -1 Falling back to public interface 2024-09-15T20:50:29.979 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:30.981 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:30.981 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:30.981 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:50:30.981 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:50:30.982 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:30.982 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:31.394 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:32.396 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:32.397 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:32.397 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:50:32.397 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:50:32.397 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:32.397 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:32.817 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:33.819 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:33.819 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:33.819 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:50:33.820 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:50:33.820 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:33.820 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:34.273 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:34.917 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:50:34.913+0000 7f0b87d947c0 -1 osd.1 0 log_to_monitors true 2024-09-15T20:50:35.274 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:35.274 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:35.275 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:50:35.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:50:35.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:35.275 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:35.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:36.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:36.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:36.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:50:36.786 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:50:36.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:36.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:37.298 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:38.300 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:38.300 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:38.300 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:50:38.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:50:38.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:38.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:38.800 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:39.804 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:39.804 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:39.804 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:50:39.805 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:50:39.805 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:39.805 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:40.376 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:41.377 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:41.377 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:41.377 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:50:41.377 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:50:41.378 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:41.378 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:41.808 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:42.809 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:42.809 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:42.809 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:50:42.810 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:50:42.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:42.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:43.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:44.321 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:44.321 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:44.321 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:50:44.321 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:50:44.321 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:44.321 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:44.794 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:45.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:45.795 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:45.796 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:50:45.796 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:50:45.796 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:45.796 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:46.239 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:47.240 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:47.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:47.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:50:47.241 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:50:47.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:47.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:47.740 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:48.741 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:48.742 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:48.742 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:50:48.742 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:50:48.742 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:48.743 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:49.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:50.585 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:50.585 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:50.585 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:50:50.586 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:50:50.586 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:50.586 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:51.359 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:52.361 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:52.361 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:52.361 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:50:52.362 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:50:52.362 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:52.362 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:53.150 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:54.152 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:54.152 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:54.152 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:50:54.152 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:50:54.153 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:54.153 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:54.891 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:55.893 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:55.893 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:55.893 INFO:tasks.workunit.client.0.smithi139.stdout:18 2024-09-15T20:50:55.894 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:50:55.894 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:55.894 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:56.703 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:57.705 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:57.705 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:57.706 INFO:tasks.workunit.client.0.smithi139.stdout:19 2024-09-15T20:50:57.706 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:50:57.706 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:57.706 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:50:58.796 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:50:59.798 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:50:59.798 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:50:59.798 INFO:tasks.workunit.client.0.smithi139.stdout:20 2024-09-15T20:50:59.799 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T20:50:59.799 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:50:59.799 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:51:00.251 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1762496741,v1:127.0.0.1:6811/1762496741] [v2:127.0.0.1:6812/1762496741,v1:127.0.0.1:6813/1762496741] exists,up a49f13f9-33e9-4f82-b5ec-e8b428f16326 2024-09-15T20:51:00.252 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:51:00.252 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:51:00.252 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:51:00.252 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-15T20:51:00.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=101214 2024-09-15T20:51:00.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2024-09-15T20:51:00.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:51:00.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:51:00.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-15T20:51:00.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:51:00.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-15T20:51:00.256 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T20:51:00.256 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:51:00.256 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:51:00.256 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:51:00.256 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-15T20:51:00.256 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-15T20:51:00.256 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:51:00.256 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:51:00.256 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T20:51:00.256 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:51:00.256 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:51:00.256 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:51:00.256 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:51:00.256 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:51:00.257 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:51:00.257 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:51:00.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:51:00.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:51:00.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:51:00.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:51:00.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T20:51:00.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T20:51:00.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:51:00.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:51:00.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:51:00.259 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:51:00.259 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:51:00.259 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:51:00.259 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-15T20:51:00.259 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:51:00.259 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=135419f4-fa78-4c50-a199-2ce283b93656 2024-09-15T20:51:00.259 INFO:tasks.workunit.client.0.smithi139.stdout:add osd2 135419f4-fa78-4c50-a199-2ce283b93656 2024-09-15T20:51:00.260 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 135419f4-fa78-4c50-a199-2ce283b93656' 2024-09-15T20:51:00.260 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:51:00.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC0SOdmaePDEBAA9SrV4pKmuf5UM/33CjlKhA== 2024-09-15T20:51:00.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC0SOdmaePDEBAA9SrV4pKmuf5UM/33CjlKhA=="}' 2024-09-15T20:51:00.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 135419f4-fa78-4c50-a199-2ce283b93656 -i td/osd-bluefs-volume-ops/2/new.json 2024-09-15T20:51:00.769 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:51:00.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2024-09-15T20:51:00.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0SOdmaePDEBAA9SrV4pKmuf5UM/33CjlKhA== --osd-uuid 135419f4-fa78-4c50-a199-2ce283b93656 2024-09-15T20:51:00.819 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:51:00.817+0000 7fb4d86b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:00.843 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:51:00.841+0000 7fb4d86b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:00.845 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:51:00.845+0000 7fb4d86b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:00.845 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:51:00.845+0000 7fb4d86b17c0 -1 bdev(0x56483bdfa800 td/osd-bluefs-volume-ops/2/block) open stat got: (1) Operation not permitted 2024-09-15T20:51:00.846 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:51:00.845+0000 7fb4d86b17c0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2024-09-15T20:51:11.624 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2024-09-15T20:51:11.624 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:51:11.625 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-15T20:51:11.625 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd2 key to auth repository 2024-09-15T20:51:11.625 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:51:12.184 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.2 2024-09-15T20:51:12.188 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-15T20:51:12.188 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:51:12.188 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:51:12.188 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:51:12.188 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:51:12.216 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:51:12.216+0000 7f92b61397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:12.222 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:51:12.220+0000 7f92b61397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:12.223 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:51:12.220+0000 7f92b61397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:12.597 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-15T20:51:12.597 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:51:12.597 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:51:12.597 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:51:12.597 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:51:12.597 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:12.597 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:51:12.598 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:51:12.598 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:12.598 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:13.026 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:14.029 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:14.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:14.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:51:14.030 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:51:14.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:14.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:14.217 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:51:14.216+0000 7f92b61397c0 -1 Falling back to public interface 2024-09-15T20:51:14.449 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:15.451 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:15.451 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:15.452 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:51:15.452 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:51:15.452 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:15.452 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:15.882 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:16.883 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:16.883 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:16.884 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:51:16.884 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:51:16.884 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:16.884 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:17.357 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:18.360 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:18.360 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:18.360 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:51:18.361 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:51:18.361 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:18.361 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:18.773 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:19.159 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:51:19.156+0000 7f92b61397c0 -1 osd.2 0 log_to_monitors true 2024-09-15T20:51:19.775 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:19.775 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:19.775 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:51:19.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:51:19.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:19.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:20.241 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:21.243 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:21.243 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:21.243 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:51:21.244 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:51:21.244 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:21.244 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:21.870 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:22.872 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:22.872 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:22.873 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:51:22.873 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:51:22.873 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:22.873 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:23.483 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:24.485 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:24.485 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:24.485 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:51:24.486 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:51:24.486 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:24.486 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:24.949 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:25.950 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:25.951 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:25.951 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:51:25.951 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:51:25.951 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:25.952 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:26.496 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:27.497 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:27.498 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:27.498 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:51:27.498 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:51:27.499 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:27.499 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:27.950 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:28.952 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:28.952 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:28.953 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:51:28.953 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:51:28.953 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:28.953 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:29.447 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:30.449 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:30.449 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:30.449 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:51:30.450 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:51:30.450 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:30.450 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:30.879 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:31.880 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:31.881 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:31.881 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:51:31.881 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:51:31.882 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:31.882 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:32.495 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:33.497 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:33.497 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:33.498 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:51:33.498 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:51:33.498 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:33.498 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:34.188 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:35.189 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:35.189 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:35.190 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:51:35.190 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:51:35.190 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:35.190 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:35.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:36.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:36.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:36.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:51:36.786 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:51:36.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:36.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:37.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:38.327 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:38.327 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:38.327 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:51:38.328 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:51:38.328 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:38.328 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:38.999 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:40.001 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:40.001 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:40.001 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:51:40.001 INFO:tasks.workunit.client.0.smithi139.stdout:18 2024-09-15T20:51:40.002 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:40.002 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:40.827 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:41.828 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:41.828 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:41.829 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:51:41.829 INFO:tasks.workunit.client.0.smithi139.stdout:19 2024-09-15T20:51:41.829 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:41.829 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:42.880 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:43.881 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:43.882 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:43.882 INFO:tasks.workunit.client.0.smithi139.stdout:20 2024-09-15T20:51:43.882 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T20:51:43.883 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:43.883 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:44.644 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:45.645 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:45.645 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:45.646 INFO:tasks.workunit.client.0.smithi139.stdout:21 2024-09-15T20:51:45.646 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-15T20:51:45.646 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:45.646 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:51:46.132 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3300998897,v1:127.0.0.1:6819/3300998897] [v2:127.0.0.1:6820/3300998897,v1:127.0.0.1:6821/3300998897] exists,up 135419f4-fa78-4c50-a199-2ce283b93656 2024-09-15T20:51:46.133 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:51:46.133 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:51:46.133 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:51:46.133 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-15T20:51:46.135 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=103794 2024-09-15T20:51:46.135 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2024-09-15T20:51:46.136 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:51:46.136 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T20:51:46.136 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-15T20:51:46.136 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T20:51:46.136 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-15T20:51:46.136 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T20:51:46.136 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:51:46.136 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:51:46.136 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:51:46.136 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-15T20:51:46.136 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-15T20:51:46.136 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T20:51:46.136 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T20:51:46.136 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T20:51:46.138 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T20:51:46.138 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:51:46.138 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:51:46.138 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:51:46.138 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:51:46.138 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:51:46.139 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:51:46.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:51:46.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:51:46.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T20:51:46.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T20:51:46.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T20:51:46.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T20:51:46.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:51:46.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:51:46.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:51:46.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:51:46.140 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T20:51:46.140 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T20:51:46.140 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-15T20:51:46.140 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T20:51:46.141 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e342e581-4cdf-4872-af3a-c5f3021ebbfa 2024-09-15T20:51:46.141 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 e342e581-4cdf-4872-af3a-c5f3021ebbfa' 2024-09-15T20:51:46.141 INFO:tasks.workunit.client.0.smithi139.stdout:add osd3 e342e581-4cdf-4872-af3a-c5f3021ebbfa 2024-09-15T20:51:46.141 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T20:51:46.163 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDiSOdm913ECRAAjWgPTnpiKCjmWOvHk2Cg0A== 2024-09-15T20:51:46.163 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDiSOdm913ECRAAjWgPTnpiKCjmWOvHk2Cg0A=="}' 2024-09-15T20:51:46.163 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e342e581-4cdf-4872-af3a-c5f3021ebbfa -i td/osd-bluefs-volume-ops/3/new.json 2024-09-15T20:51:46.675 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:51:46.690 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2024-09-15T20:51:46.691 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDiSOdm913ECRAAjWgPTnpiKCjmWOvHk2Cg0A== --osd-uuid e342e581-4cdf-4872-af3a-c5f3021ebbfa 2024-09-15T20:51:46.724 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:51:46.719+0000 7f9f25e0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:46.725 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:51:46.723+0000 7f9f25e0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:46.727 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:51:46.723+0000 7f9f25e0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:46.727 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:51:46.723+0000 7f9f25e0b7c0 -1 bdev(0x55c674121c00 td/osd-bluefs-volume-ops/3/block) open stat got: (1) Operation not permitted 2024-09-15T20:51:46.727 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:51:46.723+0000 7f9f25e0b7c0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2024-09-15T20:51:57.425 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2024-09-15T20:51:57.425 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T20:51:57.426 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd3 key to auth repository 2024-09-15T20:51:57.427 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-15T20:51:57.427 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T20:51:57.985 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.3 2024-09-15T20:51:57.985 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-15T20:51:57.985 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T20:51:57.986 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T20:51:57.986 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:51:57.986 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T20:51:58.017 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:51:58.015+0000 7ff0e46677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:58.044 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:51:58.039+0000 7ff0e46677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:58.045 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:51:58.043+0000 7ff0e46677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:51:58.424 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-15T20:51:58.424 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:51:58.424 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:51:58.424 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:51:58.424 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:51:58.424 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:51:58.425 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:58.425 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:51:58.425 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:51:58.425 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:58.906 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:51:59.852 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:51:59.847+0000 7ff0e46677c0 -1 Falling back to public interface 2024-09-15T20:51:59.908 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:51:59.909 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:51:59.913 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:51:59.913 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:51:59.914 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:51:59.914 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:00.356 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:01.357 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:01.358 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:01.358 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:52:01.358 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:52:01.358 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:01.359 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:01.853 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:02.854 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:02.854 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:02.855 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:52:02.855 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:52:02.855 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:02.855 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:03.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:04.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:04.302 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:04.303 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:52:04.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:52:04.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:04.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:04.722 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:05.347 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:52:05.342+0000 7ff0e46677c0 -1 osd.3 0 log_to_monitors true 2024-09-15T20:52:05.724 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:05.724 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:05.725 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:52:05.725 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:52:05.725 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:05.726 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:06.328 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:07.329 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:07.330 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:07.330 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:52:07.330 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:52:07.330 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:07.330 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:07.769 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:08.771 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:08.771 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:08.772 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T20:52:08.772 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T20:52:08.772 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:08.772 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:09.322 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:10.324 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:10.324 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:10.325 INFO:tasks.workunit.client.0.smithi139.stdout:8 2024-09-15T20:52:10.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-15T20:52:10.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:10.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:10.755 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:11.756 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:11.756 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:11.757 INFO:tasks.workunit.client.0.smithi139.stdout:9 2024-09-15T20:52:11.757 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-15T20:52:11.757 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:11.757 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:12.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:13.222 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:13.222 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:13.223 INFO:tasks.workunit.client.0.smithi139.stdout:10 2024-09-15T20:52:13.223 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-15T20:52:13.223 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:13.223 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:13.689 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:14.691 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:14.691 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:14.692 INFO:tasks.workunit.client.0.smithi139.stdout:11 2024-09-15T20:52:14.692 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-15T20:52:14.692 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:14.692 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:15.308 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:16.310 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:16.310 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:16.310 INFO:tasks.workunit.client.0.smithi139.stdout:12 2024-09-15T20:52:16.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-15T20:52:16.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:16.311 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:16.766 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:17.768 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:17.768 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:17.768 INFO:tasks.workunit.client.0.smithi139.stdout:13 2024-09-15T20:52:17.769 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-15T20:52:17.769 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:17.769 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:18.284 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:19.286 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:19.286 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:19.287 INFO:tasks.workunit.client.0.smithi139.stdout:14 2024-09-15T20:52:19.287 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-15T20:52:19.287 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:19.288 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:20.031 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:21.033 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:21.033 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:21.033 INFO:tasks.workunit.client.0.smithi139.stdout:15 2024-09-15T20:52:21.034 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-15T20:52:21.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:21.036 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:21.515 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:22.516 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:22.516 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:22.516 INFO:tasks.workunit.client.0.smithi139.stdout:16 2024-09-15T20:52:22.517 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-15T20:52:22.517 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:22.517 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:23.061 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:24.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:24.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:24.063 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-15T20:52:24.064 INFO:tasks.workunit.client.0.smithi139.stdout:17 2024-09-15T20:52:24.064 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:24.067 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:24.915 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:25.917 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:25.917 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:25.917 INFO:tasks.workunit.client.0.smithi139.stdout:18 2024-09-15T20:52:25.918 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-15T20:52:25.918 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:25.918 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:27.048 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:28.049 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:28.050 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:28.050 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-15T20:52:28.050 INFO:tasks.workunit.client.0.smithi139.stdout:19 2024-09-15T20:52:28.050 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:28.050 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:28.883 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:29.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:29.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:29.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-15T20:52:29.885 INFO:tasks.workunit.client.0.smithi139.stdout:20 2024-09-15T20:52:29.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:29.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:30.762 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:52:31.763 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:52:31.763 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:52:31.764 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-15T20:52:31.764 INFO:tasks.workunit.client.0.smithi139.stdout:21 2024-09-15T20:52:31.764 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:52:31.764 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:52:32.178 INFO:tasks.workunit.client.0.smithi139.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1154022732,v1:127.0.0.1:6827/1154022732] [v2:127.0.0.1:6828/1154022732,v1:127.0.0.1:6829/1154022732] exists,up e342e581-4cdf-4872-af3a-c5f3021ebbfa 2024-09-15T20:52:32.179 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:52:32.179 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:52:32.179 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:52:32.179 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-15T20:52:32.181 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=106398 2024-09-15T20:52:32.181 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2024-09-15T20:52:37.183 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2024-09-15T20:52:37.183 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2024-09-15T20:52:37.874 INFO:tasks.workunit.client.0.smithi139.stderr:pool 'foo' already exists 2024-09-15T20:52:37.891 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T20:52:38.893 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-15T20:52:39.092 INFO:tasks.workunit.client.0.smithi139.stdout:hints = 1 2024-09-15T20:52:39.092 INFO:tasks.workunit.client.0.smithi139.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-15T20:52:39.092 INFO:tasks.workunit.client.0.smithi139.stdout:Object prefix: benchmark_data_smithi139_107813 2024-09-15T20:52:39.092 INFO:tasks.workunit.client.0.smithi139.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T20:52:39.092 INFO:tasks.workunit.client.0.smithi139.stdout: 0 0 0 0 0 0 - 0 2024-09-15T20:52:40.092 INFO:tasks.workunit.client.0.smithi139.stdout: 1 16 40 24 0.0937364 0.09375 0.417068 0.616546 2024-09-15T20:52:41.092 INFO:tasks.workunit.client.0.smithi139.stdout: 2 16 80 64 0.124984 0.15625 0.203205 0.418048 2024-09-15T20:52:42.092 INFO:tasks.workunit.client.0.smithi139.stdout: 3 16 127 111 0.144512 0.183594 0.292106 0.40634 2024-09-15T20:52:43.093 INFO:tasks.workunit.client.0.smithi139.stdout: 4 16 177 161 0.157204 0.195312 0.30854 0.376746 2024-09-15T20:52:44.093 INFO:tasks.workunit.client.0.smithi139.stdout: 5 16 239 223 0.174194 0.242188 0.292144 0.352526 2024-09-15T20:52:45.093 INFO:tasks.workunit.client.0.smithi139.stdout: 6 16 292 276 0.179662 0.207031 0.283795 0.339957 2024-09-15T20:52:46.093 INFO:tasks.workunit.client.0.smithi139.stdout: 7 16 341 325 0.181335 0.191406 0.317369 0.334502 2024-09-15T20:52:47.093 INFO:tasks.workunit.client.0.smithi139.stdout: 8 16 399 383 0.186985 0.226562 0.261653 0.328389 2024-09-15T20:52:48.093 INFO:tasks.workunit.client.0.smithi139.stdout: 9 16 453 437 0.189644 0.210938 0.295449 0.322362 2024-09-15T20:52:49.093 INFO:tasks.workunit.client.0.smithi139.stdout: 10 16 511 495 0.193333 0.226562 0.366635 0.319564 2024-09-15T20:52:50.093 INFO:tasks.workunit.client.0.smithi139.stdout: 11 16 564 548 0.194577 0.207031 0.293253 0.317993 2024-09-15T20:52:51.094 INFO:tasks.workunit.client.0.smithi139.stdout: 12 16 623 607 0.197565 0.230469 0.25252 0.313521 2024-09-15T20:52:52.094 INFO:tasks.workunit.client.0.smithi139.stdout: 13 16 671 655 0.196788 0.1875 0.346984 0.312761 2024-09-15T20:52:53.094 INFO:tasks.workunit.client.0.smithi139.stdout: 14 16 726 710 0.198076 0.214844 0.225001 0.311576 2024-09-15T20:52:54.094 INFO:tasks.workunit.client.0.smithi139.stdout: 15 16 783 767 0.199712 0.222656 0.301646 0.310635 2024-09-15T20:52:55.094 INFO:tasks.workunit.client.0.smithi139.stdout: 16 16 838 822 0.200656 0.214844 0.291736 0.309111 2024-09-15T20:52:56.094 INFO:tasks.workunit.client.0.smithi139.stdout: 17 16 887 871 0.200111 0.191406 0.292074 0.308944 2024-09-15T20:52:57.094 INFO:tasks.workunit.client.0.smithi139.stdout: 18 16 943 927 0.201144 0.21875 0.294318 0.30805 2024-09-15T20:52:58.095 INFO:tasks.workunit.client.0.smithi139.stdout: 19 16 991 975 0.200424 0.1875 0.322889 0.309048 2024-09-15T20:52:59.095 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:52:59.096411+0000 min lat: 0.159865 max lat: 0.945123 avg lat: 0.30787 2024-09-15T20:52:59.095 INFO:tasks.workunit.client.0.smithi139.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T20:52:59.095 INFO:tasks.workunit.client.0.smithi139.stdout: 20 16 1047 1031 0.201339 0.21875 0.298765 0.30787 2024-09-15T20:53:00.095 INFO:tasks.workunit.client.0.smithi139.stdout: 21 16 1098 1082 0.201237 0.199219 0.291861 0.30836 2024-09-15T20:53:01.095 INFO:tasks.workunit.client.0.smithi139.stdout: 22 16 1148 1132 0.200967 0.195312 0.332351 0.309545 2024-09-15T20:53:02.095 INFO:tasks.workunit.client.0.smithi139.stdout: 23 16 1196 1180 0.20038 0.1875 0.29219 0.309918 2024-09-15T20:53:03.095 INFO:tasks.workunit.client.0.smithi139.stdout: 24 16 1252 1236 0.201145 0.21875 0.309906 0.309313 2024-09-15T20:53:04.095 INFO:tasks.workunit.client.0.smithi139.stdout: 25 16 1303 1287 0.201067 0.199219 0.284139 0.308771 2024-09-15T20:53:05.096 INFO:tasks.workunit.client.0.smithi139.stdout: 26 16 1335 1319 0.19814 0.125 0.536543 0.313651 2024-09-15T20:53:06.096 INFO:tasks.workunit.client.0.smithi139.stdout: 27 16 1388 1372 0.198468 0.207031 0.292258 0.313455 2024-09-15T20:53:07.096 INFO:tasks.workunit.client.0.smithi139.stdout: 28 16 1441 1425 0.198773 0.207031 0.28119 0.312486 2024-09-15T20:53:08.096 INFO:tasks.workunit.client.0.smithi139.stdout: 29 16 1493 1477 0.198922 0.203125 0.29238 0.312201 2024-09-15T20:53:09.096 INFO:tasks.workunit.client.0.smithi139.stdout: 30 16 1550 1534 0.199712 0.222656 0.309283 0.311544 2024-09-15T20:53:10.097 INFO:tasks.workunit.client.0.smithi139.stdout:Total time run: 30.2059 2024-09-15T20:53:10.097 INFO:tasks.workunit.client.0.smithi139.stdout:Total writes made: 1550 2024-09-15T20:53:10.097 INFO:tasks.workunit.client.0.smithi139.stdout:Write size: 4096 2024-09-15T20:53:10.097 INFO:tasks.workunit.client.0.smithi139.stdout:Object size: 4096 2024-09-15T20:53:10.097 INFO:tasks.workunit.client.0.smithi139.stdout:Bandwidth (MB/sec): 0.200447 2024-09-15T20:53:10.097 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev Bandwidth: 0.0303572 2024-09-15T20:53:10.097 INFO:tasks.workunit.client.0.smithi139.stdout:Max bandwidth (MB/sec): 0.242188 2024-09-15T20:53:10.097 INFO:tasks.workunit.client.0.smithi139.stdout:Min bandwidth (MB/sec): 0.09375 2024-09-15T20:53:10.097 INFO:tasks.workunit.client.0.smithi139.stdout:Average IOPS: 51 2024-09-15T20:53:10.097 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev IOPS: 7.77145 2024-09-15T20:53:10.098 INFO:tasks.workunit.client.0.smithi139.stdout:Max IOPS: 62 2024-09-15T20:53:10.098 INFO:tasks.workunit.client.0.smithi139.stdout:Min IOPS: 24 2024-09-15T20:53:10.098 INFO:tasks.workunit.client.0.smithi139.stdout:Average Latency(s): 0.311284 2024-09-15T20:53:10.098 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev Latency(s): 0.0740009 2024-09-15T20:53:10.098 INFO:tasks.workunit.client.0.smithi139.stdout:Max latency(s): 0.945123 2024-09-15T20:53:10.098 INFO:tasks.workunit.client.0.smithi139.stdout:Min latency(s): 0.159865 2024-09-15T20:53:10.220 INFO:tasks.workunit.client.0.smithi139.stdout:after bench 2024-09-15T20:53:10.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2024-09-15T20:53:10.220 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 98635 2024-09-15T20:53:10.221 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2024-09-15T20:53:11.222 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 98635 2024-09-15T20:53:11.222 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (98635) - No such process 2024-09-15T20:53:11.223 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2024-09-15T20:53:11.779 INFO:tasks.workunit.client.0.smithi139.stderr:osd.0 is already down. 2024-09-15T20:53:11.799 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 101214 2024-09-15T20:53:11.799 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2024-09-15T20:53:12.800 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 101214 2024-09-15T20:53:12.801 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (101214) - No such process 2024-09-15T20:53:12.801 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2024-09-15T20:53:13.249 INFO:tasks.workunit.client.0.smithi139.stderr:osd.1 is already down. 2024-09-15T20:53:13.266 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 103794 2024-09-15T20:53:13.266 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2024-09-15T20:53:14.268 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 103794 2024-09-15T20:53:14.268 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (103794) - No such process 2024-09-15T20:53:14.268 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2024-09-15T20:53:14.786 INFO:tasks.workunit.client.0.smithi139.stderr:osd.2 is already down. 2024-09-15T20:53:14.804 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 106398 2024-09-15T20:53:14.804 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2024-09-15T20:53:15.805 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 106398 2024-09-15T20:53:15.806 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (106398) - No such process 2024-09-15T20:53:15.806 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2024-09-15T20:53:16.388 INFO:tasks.workunit.client.0.smithi139.stderr:osd.3 is already down. 2024-09-15T20:53:16.406 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-15T20:53:18.781 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:53:18.807 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-15T20:53:21.189 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:53:21.214 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-15T20:53:23.601 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:53:23.627 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-15T20:53:25.993 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:53:26.017 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-09-15T20:53:26.029 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-09-15T20:53:26.057 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:53:27.264 INFO:tasks.workunit.client.0.smithi139.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-15T20:53:27.265 INFO:tasks.workunit.client.0.smithi139.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-15T20:53:27.265 INFO:tasks.workunit.client.0.smithi139.stdout:2 : device size 0x100000000 : using 0x804a2000(2.0 GiB) 2024-09-15T20:53:27.265 INFO:tasks.workunit.client.0.smithi139.stdout:Expanding DB/WAL... 2024-09-15T20:53:27.265 INFO:tasks.workunit.client.0.smithi139.stdout:0 : expanding to 0x536870912 2024-09-15T20:53:27.277 INFO:tasks.workunit.client.0.smithi139.stdout:0 : size label updated to 536870912 2024-09-15T20:53:27.277 INFO:tasks.workunit.client.0.smithi139.stdout:1 : expanding to 0x1073741824 2024-09-15T20:53:27.285 INFO:tasks.workunit.client.0.smithi139.stdout:1 : size label updated to 1073741824 2024-09-15T20:53:27.285 INFO:tasks.workunit.client.0.smithi139.stdout:2 : expanding from 0x80000000 to 0x100000000 2024-09-15T20:53:27.510 INFO:tasks.workunit.client.0.smithi139.stdout:2 : size label updated to 4294967296 2024-09-15T20:53:33.131 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2024-09-15T20:53:33.133 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2024-09-15T20:53:33.159 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:53:34.350 INFO:tasks.workunit.client.0.smithi139.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-15T20:53:34.350 INFO:tasks.workunit.client.0.smithi139.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-15T20:53:34.350 INFO:tasks.workunit.client.0.smithi139.stdout:2 : device size 0x101000000 : using 0x814d8000(2.0 GiB) 2024-09-15T20:53:34.350 INFO:tasks.workunit.client.0.smithi139.stdout:Expanding DB/WAL... 2024-09-15T20:53:34.350 INFO:tasks.workunit.client.0.smithi139.stdout:0 : expanding to 0x536870912 2024-09-15T20:53:34.367 INFO:tasks.workunit.client.0.smithi139.stdout:0 : size label updated to 536870912 2024-09-15T20:53:34.368 INFO:tasks.workunit.client.0.smithi139.stdout:1 : expanding to 0x1073741824 2024-09-15T20:53:34.376 INFO:tasks.workunit.client.0.smithi139.stdout:1 : size label updated to 1073741824 2024-09-15T20:53:34.376 INFO:tasks.workunit.client.0.smithi139.stdout:2 : expanding from 0x80000000 to 0x101000000 2024-09-15T20:53:34.526 INFO:tasks.workunit.client.0.smithi139.stdout:2 : size label updated to 4311744512 2024-09-15T20:53:39.920 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2024-09-15T20:53:39.921 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2024-09-15T20:53:39.949 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:53:41.142 INFO:tasks.workunit.client.0.smithi139.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-15T20:53:41.142 INFO:tasks.workunit.client.0.smithi139.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-15T20:53:41.142 INFO:tasks.workunit.client.0.smithi139.stdout:2 : device size 0x100020000 : using 0x80503000(2.0 GiB) 2024-09-15T20:53:41.142 INFO:tasks.workunit.client.0.smithi139.stdout:Expanding DB/WAL... 2024-09-15T20:53:41.142 INFO:tasks.workunit.client.0.smithi139.stdout:0 : expanding to 0x536870912 2024-09-15T20:53:41.159 INFO:tasks.workunit.client.0.smithi139.stdout:0 : size label updated to 536870912 2024-09-15T20:53:41.159 INFO:tasks.workunit.client.0.smithi139.stdout:1 : expanding to 0x1073741824 2024-09-15T20:53:41.168 INFO:tasks.workunit.client.0.smithi139.stdout:1 : size label updated to 1073741824 2024-09-15T20:53:41.169 INFO:tasks.workunit.client.0.smithi139.stdout:2 : expanding from 0x80000000 to 0x100020000 2024-09-15T20:53:41.326 INFO:tasks.workunit.client.0.smithi139.stdout:2 : size label updated to 4295098368 2024-09-15T20:53:47.059 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2024-09-15T20:53:47.060 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2024-09-15T20:53:47.088 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:53:48.284 INFO:tasks.workunit.client.0.smithi139.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-15T20:53:48.284 INFO:tasks.workunit.client.0.smithi139.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-15T20:53:48.284 INFO:tasks.workunit.client.0.smithi139.stdout:2 : device size 0xfff00000 : using 0x803e6000(2.0 GiB) 2024-09-15T20:53:48.284 INFO:tasks.workunit.client.0.smithi139.stdout:Expanding DB/WAL... 2024-09-15T20:53:48.285 INFO:tasks.workunit.client.0.smithi139.stdout:0 : expanding to 0x536870912 2024-09-15T20:53:48.301 INFO:tasks.workunit.client.0.smithi139.stdout:0 : size label updated to 536870912 2024-09-15T20:53:48.301 INFO:tasks.workunit.client.0.smithi139.stdout:1 : expanding to 0x1073741824 2024-09-15T20:53:48.309 INFO:tasks.workunit.client.0.smithi139.stdout:1 : size label updated to 1073741824 2024-09-15T20:53:48.310 INFO:tasks.workunit.client.0.smithi139.stdout:2 : expanding from 0x80000000 to 0xfff00000 2024-09-15T20:53:48.451 INFO:tasks.workunit.client.0.smithi139.stdout:2 : size label updated to 4293918720 2024-09-15T20:53:54.143 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-15T20:53:56.509 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:53:56.533 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-15T20:53:58.905 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:53:58.928 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-15T20:54:01.330 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:54:01.353 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-15T20:54:03.722 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:54:03.743 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-09-15T20:54:03.770 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:54:05.839 INFO:tasks.workunit.client.0.smithi139.stdout:0 : device size 0x1ffff000 : using 0x1600000(22 MiB) 2024-09-15T20:54:05.839 INFO:tasks.workunit.client.0.smithi139.stdout:1 : device size 0x3fffe000 : using 0x1600000(22 MiB) 2024-09-15T20:54:05.840 INFO:tasks.workunit.client.0.smithi139.stdout:2 : device size 0x100000000 : using 0x4a2000(4.6 MiB) 2024-09-15T20:54:05.843 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2024-09-15T20:54:05.871 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:54:08.793 INFO:tasks.workunit.client.0.smithi139.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-09-15T20:54:08.804 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-15T20:54:10.686 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:54:10.707 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-09-15T20:54:10.734 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:54:13.642 INFO:tasks.workunit.client.0.smithi139.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-15T20:54:13.648 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-15T20:54:15.514 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:54:15.535 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2024-09-15T20:54:15.562 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:54:18.490 INFO:tasks.workunit.client.0.smithi139.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-09-15T20:54:18.490 INFO:tasks.workunit.client.0.smithi139.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-09-15T20:54:18.500 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-15T20:54:19.885 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:54:19.898 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2024-09-15T20:54:19.924 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:54:20.774 INFO:tasks.workunit.client.0.smithi139.stderr:Migrate to WAL device isn't supported. 2024-09-15T20:54:20.777 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2024-09-15T20:54:20.777 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-15T20:54:23.158 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:54:23.180 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-09-15T20:54:23.208 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:54:26.362 INFO:tasks.workunit.client.0.smithi139.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-09-15T20:54:26.372 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-15T20:54:28.242 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:54:28.263 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2024-09-15T20:54:28.289 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:54:30.420 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-15T20:54:32.302 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:54:32.324 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-15T20:54:32.324 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:54:32.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:54:32.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:54:32.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:54:32.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-15T20:54:32.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T20:54:32.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:54:32.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:54:32.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:54:32.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-15T20:54:32.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-15T20:54:32.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:54:32.326 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:54:32.326 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T20:54:32.327 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:54:32.328 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:54:32.328 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:54:32.328 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:54:32.328 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:54:32.328 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:54:32.328 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:54:32.328 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:54:32.328 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:54:32.328 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T20:54:32.328 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T20:54:32.328 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:54:32.328 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:54:32.329 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:54:32.329 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:54:32.329 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:54:32.329 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:54:32.329 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-15T20:54:32.329 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T20:54:32.330 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:54:32.330 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:54:32.331 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-15T20:54:32.333 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:54:32.334 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:54:32.334 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:54:32.334 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:54:32.362 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:32.358+0000 7fc87c6667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:54:32.366 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:32.362+0000 7fc87c6667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:54:32.367 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:32.362+0000 7fc87c6667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:54:32.819 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:54:32.820 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:54:32.820 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:54:32.820 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:54:32.820 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:54:32.820 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:32.820 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:54:32.821 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:54:32.821 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:32.821 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:54:33.281 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:33.644 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:33.638+0000 7fc87c6667c0 -1 Falling back to public interface 2024-09-15T20:54:34.283 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:34.283 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:34.283 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:54:34.284 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:54:34.284 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:34.284 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:54:34.721 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:35.723 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:35.723 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:35.723 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:54:35.724 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:54:35.724 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:35.724 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:54:36.189 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:37.190 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:37.190 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:37.190 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:54:37.191 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:54:37.191 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:37.191 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:54:37.620 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:38.009 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:38.002+0000 7fc87c6667c0 -1 osd.0 28 log_to_monitors true 2024-09-15T20:54:38.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:38.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:38.621 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:54:38.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:54:38.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:38.622 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:54:39.085 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 39 up_thru 39 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/2756612130,v1:127.0.0.1:6803/2756612130] [v2:127.0.0.1:6804/2756612130,v1:127.0.0.1:6805/2756612130] exists,up 3b45a55c-3ec6-4b29-933b-e92e4449d4c3 2024-09-15T20:54:39.085 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:54:39.085 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:54:39.086 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:54:39.086 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-15T20:54:39.088 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=119957 2024-09-15T20:54:39.088 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-15T20:54:39.088 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:54:39.088 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:54:39.088 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:54:39.088 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:54:39.089 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-15T20:54:39.089 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T20:54:39.089 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:54:39.089 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:54:39.089 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:54:39.089 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-15T20:54:39.089 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-15T20:54:39.089 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:54:39.089 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:54:39.089 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T20:54:39.089 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:54:39.089 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:54:39.089 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:54:39.091 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:54:39.091 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:54:39.091 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:54:39.091 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:54:39.091 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:54:39.091 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:54:39.091 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T20:54:39.091 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T20:54:39.091 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:54:39.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:54:39.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:54:39.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:54:39.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:54:39.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:54:39.092 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-15T20:54:39.092 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.1 2024-09-15T20:54:39.093 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:54:39.093 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:54:39.093 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-15T20:54:39.093 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:54:39.093 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:54:39.093 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:54:39.094 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:54:39.122 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:39.118+0000 7efdbd3d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:54:39.132 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:39.126+0000 7efdbd3d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:54:39.133 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:39.126+0000 7efdbd3d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:54:39.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:54:39.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:54:39.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:54:39.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:54:39.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:54:39.561 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:39.562 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:54:39.562 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:54:39.562 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:39.562 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:54:39.965 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:40.865 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:40.862+0000 7efdbd3d37c0 -1 Falling back to public interface 2024-09-15T20:54:40.966 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:40.967 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:40.967 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:54:40.967 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:54:40.968 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:40.968 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:54:41.392 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:42.390 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:42.391 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:42.391 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:54:42.391 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:54:42.391 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:42.391 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:54:42.807 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:43.809 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:43.809 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:43.809 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:54:43.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:54:43.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:43.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:54:44.245 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:45.092 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:45.090+0000 7efdbd3d37c0 -1 osd.1 31 log_to_monitors true 2024-09-15T20:54:45.246 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:45.247 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:45.247 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:54:45.247 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:54:45.247 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:45.247 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:54:45.708 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 42 up_thru 29 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/807834013,v1:127.0.0.1:6811/807834013] [v2:127.0.0.1:6812/807834013,v1:127.0.0.1:6813/807834013] exists,up a49f13f9-33e9-4f82-b5ec-e8b428f16326 2024-09-15T20:54:45.708 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:54:45.708 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:54:45.709 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:54:45.709 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-15T20:54:45.714 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=120930 2024-09-15T20:54:45.714 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-15T20:54:45.715 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:54:45.715 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:54:45.715 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T20:54:45.715 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:54:45.715 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-15T20:54:45.715 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T20:54:45.715 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:54:45.715 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:54:45.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:54:45.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-15T20:54:45.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-15T20:54:45.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:54:45.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:54:45.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T20:54:45.716 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:54:45.716 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:54:45.717 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:54:45.717 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:54:45.717 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:54:45.717 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:54:45.717 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:54:45.717 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:54:45.717 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:54:45.717 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T20:54:45.718 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T20:54:45.718 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:54:45.718 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:54:45.718 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:54:45.718 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:54:45.718 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:54:45.718 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:54:45.718 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-15T20:54:45.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T20:54:45.719 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.2 2024-09-15T20:54:45.720 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:54:45.720 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-15T20:54:45.720 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T20:54:45.720 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:54:45.720 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:54:45.720 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:54:45.744 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:45.742+0000 7f2f4cb277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:54:45.746 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:45.746+0000 7f2f4cb277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:54:45.747 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:45.746+0000 7f2f4cb277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:54:46.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T20:54:46.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:54:46.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:54:46.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:54:46.303 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:54:46.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:54:46.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:46.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:54:46.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:54:46.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:46.867 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:47.541 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:47.538+0000 7f2f4cb277c0 -1 Falling back to public interface 2024-09-15T20:54:47.869 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:47.869 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:47.869 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:54:47.870 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:54:47.870 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:47.870 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:54:48.299 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:49.300 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:54:49.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:49.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:49.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:54:49.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:49.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:54:49.773 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:50.737 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:50.734+0000 7f2f4cb277c0 -1 osd.2 34 log_to_monitors true 2024-09-15T20:54:50.774 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:50.774 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:50.775 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:54:50.775 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:54:50.775 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:54:50.775 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:51.231 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up in weight 1 up_from 46 up_thru 32 down_at 35 last_clean_interval [15,34) [v2:127.0.0.1:6818/642948372,v1:127.0.0.1:6819/642948372] [v2:127.0.0.1:6820/642948372,v1:127.0.0.1:6821/642948372] exists,up 135419f4-fa78-4c50-a199-2ce283b93656 2024-09-15T20:54:51.231 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:54:51.231 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:54:51.231 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:54:51.232 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-15T20:54:51.234 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=121903 2024-09-15T20:54:51.234 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-15T20:54:51.234 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:54:51.234 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:54:51.234 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T20:54:51.234 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:54:51.234 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-15T20:54:51.234 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T20:54:51.234 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:54:51.235 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:54:51.235 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:54:51.235 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-15T20:54:51.235 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-15T20:54:51.235 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:54:51.235 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:54:51.235 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T20:54:51.235 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:54:51.235 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:54:51.235 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:54:51.237 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:54:51.237 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:54:51.237 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:54:51.237 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:54:51.237 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:54:51.237 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:54:51.238 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T20:54:51.238 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T20:54:51.238 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:54:51.238 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:54:51.238 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:54:51.238 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:54:51.238 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:54:51.238 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:54:51.238 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-15T20:54:51.238 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T20:54:51.239 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.3 2024-09-15T20:54:51.239 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:54:51.239 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-15T20:54:51.239 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T20:54:51.239 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:54:51.240 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:54:51.240 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:54:51.263 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:51.262+0000 7f4099e307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:54:51.265 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:51.262+0000 7f4099e307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:54:51.267 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:51.266+0000 7f4099e307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:54:52.095 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T20:54:52.095 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:54:52.095 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:54:52.095 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:54:52.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:54:52.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:52.096 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:54:52.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:54:52.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:52.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:52.784 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:52.828 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:52.826+0000 7f4099e307c0 -1 Falling back to public interface 2024-09-15T20:54:53.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:53.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:53.785 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:54:53.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:54:53.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:53.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:54.266 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:55.267 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:54:55.268 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:55.268 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:55.268 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:54:55.268 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:55.268 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:55.785 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:56.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:56.787 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:56.787 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:54:56.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:54:56.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:56.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:56.972 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:54:56.970+0000 7f4099e307c0 -1 osd.3 36 log_to_monitors true 2024-09-15T20:54:57.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:54:58.305 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:54:58.305 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:54:58.305 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:54:58.305 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:54:58.305 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:54:58.306 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:54:58.962 INFO:tasks.workunit.client.0.smithi139.stdout:osd.3 up in weight 1 up_from 50 up_thru 50 down_at 37 last_clean_interval [20,36) [v2:127.0.0.1:6826/3309092450,v1:127.0.0.1:6827/3309092450] [v2:127.0.0.1:6828/3309092450,v1:127.0.0.1:6829/3309092450] exists,up e342e581-4cdf-4872-af3a-c5f3021ebbfa 2024-09-15T20:54:58.963 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:54:58.963 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:54:58.963 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:54:58.963 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-15T20:54:58.964 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=122846 2024-09-15T20:54:58.964 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2024-09-15T20:54:58.964 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:54:58.964 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:54:58.964 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:54:58.966 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:54:58.966 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:54:58.966 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:54:58.966 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:54:58.966 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:54:58.966 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:54:59.069 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:54:59.069 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:54:59.069 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:54:59.070 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:54:59.070 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:54:59.070 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:54:59.676 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:54:59.676 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:54:59.676 INFO:tasks.workunit.client.0.smithi139.stderr:2 2024-09-15T20:54:59.676 INFO:tasks.workunit.client.0.smithi139.stderr:3' 2024-09-15T20:54:59.676 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:54:59.676 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:54:59.677 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:54:59.867 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=167503724551 2024-09-15T20:54:59.867 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 167503724551 2024-09-15T20:54:59.867 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-167503724551' 2024-09-15T20:54:59.867 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:54:59.867 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:55:00.015 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=180388626437 2024-09-15T20:55:00.016 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 180388626437 2024-09-15T20:55:00.016 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-167503724551 1-180388626437' 2024-09-15T20:55:00.016 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:00.016 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:55:00.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=197568495620 2024-09-15T20:55:00.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 197568495620 2024-09-15T20:55:00.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-167503724551 1-180388626437 2-197568495620' 2024-09-15T20:55:00.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:55:00.157 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:55:00.305 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364803 2024-09-15T20:55:00.305 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364803 2024-09-15T20:55:00.305 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-167503724551 1-180388626437 2-197568495620 3-214748364803' 2024-09-15T20:55:00.305 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:00.306 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-167503724551 2024-09-15T20:55:00.306 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:00.307 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:55:00.308 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-167503724551 2024-09-15T20:55:00.308 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:00.309 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=167503724551 2024-09-15T20:55:00.309 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 167503724551' 2024-09-15T20:55:00.309 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 167503724551 2024-09-15T20:55:00.310 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:55:00.738 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724549 -lt 167503724551 2024-09-15T20:55:00.738 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T20:55:01.739 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T20:55:01.740 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:55:02.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724551 -lt 167503724551 2024-09-15T20:55:02.157 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:02.158 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:02.158 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-180388626437 2024-09-15T20:55:02.160 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:55:02.161 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-180388626437 2024-09-15T20:55:02.161 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:02.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=180388626437 2024-09-15T20:55:02.162 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.1 seq 180388626437 2024-09-15T20:55:02.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 180388626437' 2024-09-15T20:55:02.162 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:55:02.648 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 180388626438 -lt 180388626437 2024-09-15T20:55:02.648 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:02.649 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-197568495620 2024-09-15T20:55:02.649 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:02.650 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:55:02.651 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-197568495620 2024-09-15T20:55:02.651 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:02.652 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=197568495620 2024-09-15T20:55:02.652 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 197568495620 2024-09-15T20:55:02.653 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 197568495620' 2024-09-15T20:55:02.653 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:55:03.097 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 197568495621 -lt 197568495620 2024-09-15T20:55:03.098 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:55:03.098 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-214748364803 2024-09-15T20:55:03.098 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:55:03.100 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:55:03.101 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-214748364803 2024-09-15T20:55:03.101 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:55:03.102 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364803 2024-09-15T20:55:03.102 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 214748364803' 2024-09-15T20:55:03.102 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.3 seq 214748364803 2024-09-15T20:55:03.103 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:55:03.547 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364804 -lt 214748364803 2024-09-15T20:55:03.547 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T20:55:03.548 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:55:03.548 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:55:04.158 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-15T20:55:04.158 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T20:55:04.159 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T20:55:04.159 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T20:55:04.159 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T20:55:04.159 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T20:55:04.159 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T20:55:04.160 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T20:55:04.574 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-15T20:55:04.574 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T20:55:04.575 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T20:55:04.575 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T20:55:05.144 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-15T20:55:05.144 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T20:55:05.144 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T20:55:05.144 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-15T20:55:05.186 INFO:tasks.workunit.client.0.smithi139.stdout:hints = 1 2024-09-15T20:55:05.186 INFO:tasks.workunit.client.0.smithi139.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-15T20:55:05.186 INFO:tasks.workunit.client.0.smithi139.stdout:Object prefix: benchmark_data_smithi139_124190 2024-09-15T20:55:05.186 INFO:tasks.workunit.client.0.smithi139.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T20:55:05.186 INFO:tasks.workunit.client.0.smithi139.stdout: 0 0 0 0 0 0 - 0 2024-09-15T20:55:06.186 INFO:tasks.workunit.client.0.smithi139.stdout: 1 16 59 43 0.167958 0.167969 0.216012 0.309369 2024-09-15T20:55:07.188 INFO:tasks.workunit.client.0.smithi139.stdout: 2 16 110 94 0.183578 0.199219 0.330287 0.314559 2024-09-15T20:55:08.186 INFO:tasks.workunit.client.0.smithi139.stdout: 3 16 162 146 0.190087 0.203125 0.356688 0.314936 2024-09-15T20:55:09.187 INFO:tasks.workunit.client.0.smithi139.stdout: 4 16 210 194 0.189434 0.1875 0.178878 0.318268 2024-09-15T20:55:10.187 INFO:tasks.workunit.client.0.smithi139.stdout: 5 16 263 247 0.192949 0.207031 0.289616 0.316727 2024-09-15T20:55:11.187 INFO:tasks.workunit.client.0.smithi139.stdout: 6 16 310 294 0.191387 0.183594 0.356357 0.317287 2024-09-15T20:55:12.187 INFO:tasks.workunit.client.0.smithi139.stdout: 7 16 361 345 0.192503 0.199219 0.33048 0.317344 2024-09-15T20:55:13.187 INFO:tasks.workunit.client.0.smithi139.stdout: 8 16 416 400 0.195293 0.214844 0.268817 0.314384 2024-09-15T20:55:14.187 INFO:tasks.workunit.client.0.smithi139.stdout: 9 16 457 441 0.191387 0.160156 0.366758 0.315229 2024-09-15T20:55:15.187 INFO:tasks.workunit.client.0.smithi139.stdout: 10 16 513 497 0.19412 0.21875 0.230518 0.317758 2024-09-15T20:55:16.187 INFO:tasks.workunit.client.0.smithi139.stdout: 11 16 565 549 0.194936 0.203125 0.283873 0.317227 2024-09-15T20:55:17.187 INFO:tasks.workunit.client.0.smithi139.stdout: 12 16 625 609 0.19822 0.234375 0.212152 0.312177 2024-09-15T20:55:18.188 INFO:tasks.workunit.client.0.smithi139.stdout: 13 16 672 656 0.197093 0.183594 0.292224 0.311897 2024-09-15T20:55:19.188 INFO:tasks.workunit.client.0.smithi139.stdout: 14 16 724 708 0.197523 0.203125 0.316621 0.311208 2024-09-15T20:55:20.188 INFO:tasks.workunit.client.0.smithi139.stdout: 15 16 770 754 0.196331 0.179688 0.247536 0.31389 2024-09-15T20:55:21.188 INFO:tasks.workunit.client.0.smithi139.stdout: 16 16 824 808 0.197242 0.210938 0.257306 0.311885 2024-09-15T20:55:22.188 INFO:tasks.workunit.client.0.smithi139.stdout: 17 16 877 861 0.197816 0.207031 0.328221 0.313608 2024-09-15T20:55:23.188 INFO:tasks.workunit.client.0.smithi139.stdout: 18 16 929 913 0.19811 0.203125 0.327373 0.312353 2024-09-15T20:55:24.189 INFO:tasks.workunit.client.0.smithi139.stdout: 19 16 971 955 0.196317 0.164062 0.285179 0.315023 2024-09-15T20:55:25.189 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:55:25.189293+0000 min lat: 0.141443 max lat: 0.484668 avg lat: 0.311449 2024-09-15T20:55:25.189 INFO:tasks.workunit.client.0.smithi139.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T20:55:25.189 INFO:tasks.workunit.client.0.smithi139.stdout: 20 16 1038 1022 0.199585 0.261719 0.242141 0.311449 2024-09-15T20:55:26.189 INFO:tasks.workunit.client.0.smithi139.stdout: 21 16 1100 1084 0.201612 0.242188 0.240504 0.308073 2024-09-15T20:55:27.189 INFO:tasks.workunit.client.0.smithi139.stdout: 22 16 1167 1151 0.204342 0.261719 0.264944 0.303994 2024-09-15T20:55:28.189 INFO:tasks.workunit.client.0.smithi139.stdout: 23 16 1231 1215 0.206326 0.25 0.194776 0.30146 2024-09-15T20:55:29.189 INFO:tasks.workunit.client.0.smithi139.stdout: 24 16 1290 1274 0.207331 0.230469 0.230497 0.299804 2024-09-15T20:55:30.189 INFO:tasks.workunit.client.0.smithi139.stdout: 25 16 1351 1335 0.208568 0.238281 0.278061 0.29837 2024-09-15T20:55:31.190 INFO:tasks.workunit.client.0.smithi139.stdout: 26 16 1405 1389 0.208658 0.210938 0.316959 0.297293 2024-09-15T20:55:32.190 INFO:tasks.workunit.client.0.smithi139.stdout: 27 16 1470 1454 0.210332 0.253906 0.210891 0.29619 2024-09-15T20:55:33.190 INFO:tasks.workunit.client.0.smithi139.stdout: 28 16 1527 1511 0.210771 0.222656 0.222705 0.29496 2024-09-15T20:55:34.190 INFO:tasks.workunit.client.0.smithi139.stdout: 29 16 1591 1575 0.212123 0.25 0.191686 0.292964 2024-09-15T20:55:35.190 INFO:tasks.workunit.client.0.smithi139.stdout: 30 16 1646 1630 0.212213 0.214844 0.231973 0.293182 2024-09-15T20:55:36.190 INFO:tasks.workunit.client.0.smithi139.stdout:Total time run: 30.1835 2024-09-15T20:55:36.191 INFO:tasks.workunit.client.0.smithi139.stdout:Total writes made: 1646 2024-09-15T20:55:36.191 INFO:tasks.workunit.client.0.smithi139.stdout:Write size: 4096 2024-09-15T20:55:36.191 INFO:tasks.workunit.client.0.smithi139.stdout:Object size: 4096 2024-09-15T20:55:36.191 INFO:tasks.workunit.client.0.smithi139.stdout:Bandwidth (MB/sec): 0.21302 2024-09-15T20:55:36.191 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev Bandwidth: 0.0282119 2024-09-15T20:55:36.191 INFO:tasks.workunit.client.0.smithi139.stdout:Max bandwidth (MB/sec): 0.261719 2024-09-15T20:55:36.191 INFO:tasks.workunit.client.0.smithi139.stdout:Min bandwidth (MB/sec): 0.160156 2024-09-15T20:55:36.191 INFO:tasks.workunit.client.0.smithi139.stdout:Average IOPS: 54 2024-09-15T20:55:36.191 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev IOPS: 7.22225 2024-09-15T20:55:36.191 INFO:tasks.workunit.client.0.smithi139.stdout:Max IOPS: 67 2024-09-15T20:55:36.191 INFO:tasks.workunit.client.0.smithi139.stdout:Min IOPS: 41 2024-09-15T20:55:36.192 INFO:tasks.workunit.client.0.smithi139.stdout:Average Latency(s): 0.292711 2024-09-15T20:55:36.192 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev Latency(s): 0.0611804 2024-09-15T20:55:36.192 INFO:tasks.workunit.client.0.smithi139.stdout:Max latency(s): 0.484668 2024-09-15T20:55:36.192 INFO:tasks.workunit.client.0.smithi139.stdout:Min latency(s): 0.141091 2024-09-15T20:55:36.268 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 119957 2024-09-15T20:55:36.268 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2024-09-15T20:55:37.269 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 119957 2024-09-15T20:55:37.269 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (119957) - No such process 2024-09-15T20:55:37.269 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2024-09-15T20:55:37.858 INFO:tasks.workunit.client.0.smithi139.stderr:osd.0 is already down. 2024-09-15T20:55:37.878 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 120930 2024-09-15T20:55:37.878 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2024-09-15T20:55:38.880 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 120930 2024-09-15T20:55:38.880 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (120930) - No such process 2024-09-15T20:55:38.880 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2024-09-15T20:55:39.346 INFO:tasks.workunit.client.0.smithi139.stderr:osd.1 is already down. 2024-09-15T20:55:39.362 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 121903 2024-09-15T20:55:39.362 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2024-09-15T20:55:40.364 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 121903 2024-09-15T20:55:40.364 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (121903) - No such process 2024-09-15T20:55:40.364 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2024-09-15T20:55:40.793 INFO:tasks.workunit.client.0.smithi139.stderr:osd.2 is already down. 2024-09-15T20:55:40.809 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 122846 2024-09-15T20:55:40.809 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2024-09-15T20:55:41.811 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 122846 2024-09-15T20:55:41.811 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (122846) - No such process 2024-09-15T20:55:41.811 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2024-09-15T20:55:42.378 INFO:tasks.workunit.client.0.smithi139.stderr:osd.3 is already down. 2024-09-15T20:55:42.395 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-15T20:55:44.262 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:55:44.282 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2024-09-15T20:55:44.589 INFO:tasks.workunit.client.0.smithi139.stderr:512+0 records in 2024-09-15T20:55:44.589 INFO:tasks.workunit.client.0.smithi139.stderr:512+0 records out 2024-09-15T20:55:44.589 INFO:tasks.workunit.client.0.smithi139.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.304742 s, 1.8 GB/s 2024-09-15T20:55:44.589 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2024-09-15T20:55:44.612 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:55:50.498 INFO:tasks.workunit.client.0.smithi139.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2024-09-15T20:55:50.504 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-15T20:55:52.874 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:55:52.892 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-15T20:55:54.770 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:55:54.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2024-09-15T20:55:55.353 INFO:tasks.workunit.client.0.smithi139.stderr:1024+0 records in 2024-09-15T20:55:55.353 INFO:tasks.workunit.client.0.smithi139.stderr:1024+0 records out 2024-09-15T20:55:55.353 INFO:tasks.workunit.client.0.smithi139.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.565721 s, 1.9 GB/s 2024-09-15T20:55:55.353 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2024-09-15T20:55:55.376 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:56:04.022 INFO:tasks.workunit.client.0.smithi139.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2024-09-15T20:56:04.026 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2024-09-15T20:56:04.054 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:56:05.215 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:05.212+0000 7f431d48cac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:56:05.991 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:05.988+0000 7f431d48cac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:56:07.223 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-15T20:56:07.828 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:07.824+0000 7f3d63ebbac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:56:08.611 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:08.608+0000 7f3d63ebbac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:56:09.610 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:56:09.628 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-15T20:56:11.006 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:56:11.016 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2024-09-15T20:56:11.042 INFO:tasks.workunit.client.0.smithi139.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-15T20:56:11.043 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:56:12.386 INFO:tasks.workunit.client.0.smithi139.stdout:DB device added 2024-09-15T20:56:12.390 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2024-09-15T20:56:12.417 INFO:tasks.workunit.client.0.smithi139.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-15T20:56:12.417 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:56:13.271 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:13.268+0000 7fd915398ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:56:13.802 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:13.796+0000 7fd915398ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:56:13.871 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:13.868+0000 7fd915398ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:56:14.558 INFO:tasks.workunit.client.0.smithi139.stdout:WAL device added 2024-09-15T20:56:14.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2024-09-15T20:56:14.591 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:56:15.740 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:15.736+0000 7f7df9acaac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:56:16.523 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:16.520+0000 7f7df9acaac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:56:17.776 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-15T20:56:18.368 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:18.363+0000 7f07d49e7ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:56:19.151 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:19.147+0000 7f07d49e7ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:56:20.142 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:56:20.161 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-15T20:56:22.038 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:56:22.055 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2024-09-15T20:56:22.081 INFO:tasks.workunit.client.0.smithi139.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-15T20:56:22.081 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:56:24.218 INFO:tasks.workunit.client.0.smithi139.stdout:WAL device added 2024-09-15T20:56:24.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-09-15T20:56:24.249 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:56:27.402 INFO:tasks.workunit.client.0.smithi139.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2024-09-15T20:56:27.412 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-15T20:56:29.302 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:56:29.321 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-15T20:56:29.321 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:56:29.321 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:56:29.321 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:56:29.321 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:56:29.321 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-15T20:56:29.321 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T20:56:29.322 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:56:29.322 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:56:29.322 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:56:29.322 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-15T20:56:29.322 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-15T20:56:29.322 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:56:29.322 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:56:29.322 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T20:56:29.323 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:56:29.323 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:56:29.323 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:56:29.323 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:56:29.323 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:56:29.323 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:56:29.324 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:56:29.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:56:29.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:56:29.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T20:56:29.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T20:56:29.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:56:29.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:56:29.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:56:29.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:56:29.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:56:29.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:56:29.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-15T20:56:29.326 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T20:56:29.327 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:56:29.327 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:56:29.327 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-15T20:56:29.329 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:56:29.330 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:56:29.330 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:56:29.330 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:56:29.359 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:29.355+0000 7fa6f07037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:29.362 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:29.355+0000 7fa6f07037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:29.363 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:29.359+0000 7fa6f07037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:29.792 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:56:29.792 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:56:29.793 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:56:29.793 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:56:29.793 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:56:29.793 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:56:29.793 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:29.794 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:56:29.794 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:29.794 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:56:30.292 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:31.101 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:31.095+0000 7fa6f07037c0 -1 Falling back to public interface 2024-09-15T20:56:31.294 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:31.294 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:31.295 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:56:31.295 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:56:31.295 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:31.295 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:56:31.742 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:32.743 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:32.743 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:32.743 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:56:32.744 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:56:32.744 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:32.744 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:56:33.259 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:34.261 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:34.261 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:34.261 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:56:34.262 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:56:34.262 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:34.262 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:56:34.728 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:35.730 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:35.731 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:35.731 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:56:35.731 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:56:35.731 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:35.731 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:56:36.145 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:36.216 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:36.211+0000 7fa6f07037c0 -1 osd.0 53 log_to_monitors true 2024-09-15T20:56:37.147 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:37.147 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:37.147 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:56:37.148 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:56:37.148 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:37.148 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:56:37.614 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 64 up_thru 64 down_at 54 last_clean_interval [39,53) [v2:127.0.0.1:6802/1513604729,v1:127.0.0.1:6803/1513604729] [v2:127.0.0.1:6804/1513604729,v1:127.0.0.1:6805/1513604729] exists,up 3b45a55c-3ec6-4b29-933b-e92e4449d4c3 2024-09-15T20:56:37.614 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:56:37.614 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:56:37.614 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:56:37.615 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-15T20:56:37.616 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=130342 2024-09-15T20:56:37.617 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-15T20:56:37.617 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:56:37.617 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:56:37.617 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:56:37.617 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:56:37.617 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-15T20:56:37.617 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T20:56:37.617 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:56:37.617 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:56:37.617 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:56:37.617 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-15T20:56:37.617 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-15T20:56:37.618 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:56:37.618 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:56:37.618 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T20:56:37.618 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:56:37.618 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:56:37.618 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:56:37.618 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:56:37.618 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:56:37.618 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:56:37.618 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:56:37.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:56:37.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:56:37.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T20:56:37.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T20:56:37.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:56:37.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:56:37.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:56:37.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:56:37.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:56:37.620 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:56:37.620 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-15T20:56:37.620 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.1 2024-09-15T20:56:37.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:56:37.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:56:37.621 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-15T20:56:37.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:56:37.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:56:37.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:56:37.621 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:56:37.649 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:37.643+0000 7fc7a85ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:37.652 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:37.647+0000 7fc7a85ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:37.653 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:37.647+0000 7fc7a85ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:38.078 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:56:38.079 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:56:38.079 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:56:38.079 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:56:38.079 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:56:38.079 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:38.080 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:56:38.080 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:56:38.080 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:38.081 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:38.507 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:39.389 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:39.383+0000 7fc7a85ca7c0 -1 Falling back to public interface 2024-09-15T20:56:39.508 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:39.509 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:39.509 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:56:39.509 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:56:39.509 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:39.509 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:39.938 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:40.939 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:40.939 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:40.939 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:56:40.939 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:56:40.940 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:40.940 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:41.360 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:41.538 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:41.531+0000 7fc7a85ca7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:56:42.323 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:42.319+0000 7fc7a85ca7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:56:42.362 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:42.362 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:42.362 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:56:42.363 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:56:42.363 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:42.363 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:42.801 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:43.352 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:43.347+0000 7fc7a85ca7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:56:43.803 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:43.803 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:56:43.804 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:43.804 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:56:43.804 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:43.804 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:44.146 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:44.139+0000 7fc7a85ca7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:56:44.244 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:44.677 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:44.671+0000 7fc7a85ca7c0 -1 osd.1 56 log_to_monitors true 2024-09-15T20:56:45.245 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:45.245 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:56:45.245 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:45.245 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:56:45.246 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:45.246 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:56:45.981 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 67 up_thru 67 down_at 57 last_clean_interval [42,56) [v2:127.0.0.1:6810/2261506729,v1:127.0.0.1:6811/2261506729] [v2:127.0.0.1:6812/2261506729,v1:127.0.0.1:6813/2261506729] exists,up a49f13f9-33e9-4f82-b5ec-e8b428f16326 2024-09-15T20:56:45.982 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:56:45.982 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:56:45.982 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:56:45.982 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-15T20:56:45.984 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=131345 2024-09-15T20:56:45.984 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-15T20:56:45.984 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:56:45.984 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:56:45.984 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T20:56:45.984 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:56:45.985 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-15T20:56:45.985 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T20:56:45.985 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:56:45.985 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:56:45.985 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:56:45.985 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-15T20:56:45.985 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-15T20:56:45.985 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:56:45.985 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:56:45.985 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T20:56:45.987 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:56:45.987 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:56:45.987 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:56:45.987 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:56:45.987 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:56:45.987 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:56:45.987 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:56:45.987 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:56:45.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:56:45.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T20:56:45.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T20:56:45.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:56:45.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:56:45.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:56:45.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:56:45.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:56:45.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:56:45.988 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-15T20:56:45.989 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.2 2024-09-15T20:56:45.989 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T20:56:45.989 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:56:45.989 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-15T20:56:45.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T20:56:45.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:56:45.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:56:45.990 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:56:46.019 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:46.015+0000 7f98b89547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:46.117 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:46.111+0000 7f98b89547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:46.119 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:46.115+0000 7f98b89547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:46.450 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T20:56:46.450 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:56:46.450 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:56:46.450 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:56:46.450 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:56:46.450 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:46.451 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:56:46.451 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:56:46.451 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:46.451 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:56:46.901 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:47.903 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:47.903 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:47.903 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:56:47.903 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:56:47.904 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:47.904 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:56:47.949 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:47.943+0000 7f98b89547c0 -1 Falling back to public interface 2024-09-15T20:56:48.340 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:49.342 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:49.342 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:49.342 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:56:49.342 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:56:49.343 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:49.343 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:56:49.821 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:49.815+0000 7f98b89547c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:56:49.849 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:50.603 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:50.599+0000 7f98b89547c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:56:50.850 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:50.851 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:50.851 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:56:50.851 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:56:50.851 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:50.852 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:56:51.264 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:51.634 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:51.627+0000 7f98b89547c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:56:52.266 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:52.266 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:52.266 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:56:52.267 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:56:52.267 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:52.267 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:56:52.429 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:52.423+0000 7f98b89547c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:56:52.709 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:53.674 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:53.671+0000 7f98b89547c0 -1 osd.2 59 log_to_monitors true 2024-09-15T20:56:53.711 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:53.711 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:53.711 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:56:53.711 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:56:53.712 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:53.712 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:56:54.301 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:55.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:55.303 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:55.303 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T20:56:55.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T20:56:55.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:55.304 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:56:56.020 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up in weight 1 up_from 71 up_thru 71 down_at 60 last_clean_interval [46,59) [v2:127.0.0.1:6818/1438635099,v1:127.0.0.1:6819/1438635099] [v2:127.0.0.1:6820/1438635099,v1:127.0.0.1:6821/1438635099] exists,up 135419f4-fa78-4c50-a199-2ce283b93656 2024-09-15T20:56:56.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:56:56.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:56:56.020 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:56:56.021 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-15T20:56:56.023 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=132365 2024-09-15T20:56:56.023 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-15T20:56:56.023 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:56:56.023 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:56:56.023 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T20:56:56.023 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:56:56.023 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-15T20:56:56.023 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T20:56:56.023 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:56:56.023 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:56:56.023 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:56:56.023 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-15T20:56:56.023 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-15T20:56:56.024 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:56:56.024 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:56:56.024 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T20:56:56.024 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:56:56.024 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:56:56.024 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:56:56.025 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:56:56.025 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:56:56.025 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:56:56.025 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:56:56.025 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:56:56.025 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:56:56.025 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T20:56:56.026 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T20:56:56.026 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:56:56.026 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:56:56.026 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:56:56.026 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:56:56.026 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:56:56.026 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:56:56.026 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-15T20:56:56.026 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.3 2024-09-15T20:56:56.027 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T20:56:56.027 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:56:56.027 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-15T20:56:56.027 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T20:56:56.028 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:56:56.028 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:56:56.028 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:56:56.052 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:56.047+0000 7ff2193a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:56.055 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:56.051+0000 7ff2193a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:56.056 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:56.051+0000 7ff2193a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:56:56.485 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T20:56:56.485 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:56:56.486 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:56:56.486 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:56:56.486 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:56:56.486 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:56.486 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:56:56.486 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:56:56.487 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:56.487 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:56:56.932 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:57.297 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:56:57.291+0000 7ff2193a77c0 -1 Falling back to public interface 2024-09-15T20:56:57.932 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:57.932 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:57.933 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:56:57.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:56:57.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:57.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:56:58.346 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:56:59.347 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:56:59.348 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:56:59.348 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:56:59.348 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:56:59.348 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:56:59.349 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:56:59.823 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:00.825 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:00.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:00.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:57:00.826 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:57:00.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:00.826 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:57:01.248 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:57:01.999 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:57:01.994+0000 7ff2193a77c0 -1 osd.3 61 log_to_monitors true 2024-09-15T20:57:02.249 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:57:02.252 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:57:02.252 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:57:02.253 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:57:02.253 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:57:02.253 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:57:02.738 INFO:tasks.workunit.client.0.smithi139.stdout:osd.3 up in weight 1 up_from 75 up_thru 60 down_at 62 last_clean_interval [50,61) [v2:127.0.0.1:6826/188323294,v1:127.0.0.1:6827/188323294] [v2:127.0.0.1:6828/188323294,v1:127.0.0.1:6829/188323294] exists,up e342e581-4cdf-4872-af3a-c5f3021ebbfa 2024-09-15T20:57:02.738 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:57:02.738 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:57:02.738 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:57:02.738 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-15T20:57:02.739 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=133376 2024-09-15T20:57:02.739 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-15T20:57:02.776 INFO:tasks.workunit.client.0.smithi139.stdout:hints = 1 2024-09-15T20:57:02.776 INFO:tasks.workunit.client.0.smithi139.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-15T20:57:02.776 INFO:tasks.workunit.client.0.smithi139.stdout:Object prefix: benchmark_data_smithi139_134334 2024-09-15T20:57:02.776 INFO:tasks.workunit.client.0.smithi139.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T20:57:02.776 INFO:tasks.workunit.client.0.smithi139.stdout: 0 0 0 0 0 0 - 0 2024-09-15T20:57:03.776 INFO:tasks.workunit.client.0.smithi139.stdout: 1 16 22 6 0.0234349 0.0234375 0.333298 0.302876 2024-09-15T20:57:04.776 INFO:tasks.workunit.client.0.smithi139.stdout: 2 16 25 9 0.0175758 0.0117188 1.63767 0.607348 2024-09-15T20:57:05.776 INFO:tasks.workunit.client.0.smithi139.stdout: 3 16 79 63 0.0820196 0.210938 0.249313 0.709644 2024-09-15T20:57:06.776 INFO:tasks.workunit.client.0.smithi139.stdout: 4 16 134 118 0.115219 0.214844 0.220775 0.528504 2024-09-15T20:57:07.777 INFO:tasks.workunit.client.0.smithi139.stdout: 5 16 176 160 0.124984 0.164062 0.237016 0.474623 2024-09-15T20:57:08.777 INFO:tasks.workunit.client.0.smithi139.stdout: 6 16 228 212 0.138002 0.203125 0.217092 0.437721 2024-09-15T20:57:09.777 INFO:tasks.workunit.client.0.smithi139.stdout: 7 16 280 264 0.147301 0.203125 0.208385 0.414558 2024-09-15T20:57:10.777 INFO:tasks.workunit.client.0.smithi139.stdout: 8 16 332 316 0.154276 0.203125 0.341868 0.399023 2024-09-15T20:57:11.777 INFO:tasks.workunit.client.0.smithi139.stdout: 9 16 373 357 0.154927 0.160156 0.364036 0.391168 2024-09-15T20:57:12.777 INFO:tasks.workunit.client.0.smithi139.stdout: 10 16 420 404 0.157791 0.183594 0.225956 0.39147 2024-09-15T20:57:13.778 INFO:tasks.workunit.client.0.smithi139.stdout: 11 16 465 449 0.159424 0.175781 0.330497 0.386649 2024-09-15T20:57:14.778 INFO:tasks.workunit.client.0.smithi139.stdout: 12 16 513 497 0.161761 0.1875 0.208599 0.380137 2024-09-15T20:57:15.778 INFO:tasks.workunit.client.0.smithi139.stdout: 13 16 563 547 0.16434 0.195312 0.213395 0.376209 2024-09-15T20:57:16.778 INFO:tasks.workunit.client.0.smithi139.stdout: 14 16 612 596 0.166272 0.191406 0.333497 0.372283 2024-09-15T20:57:17.778 INFO:tasks.workunit.client.0.smithi139.stdout: 15 16 645 629 0.16378 0.128906 0.615954 0.374723 2024-09-15T20:57:18.778 INFO:tasks.workunit.client.0.smithi139.stdout: 16 16 690 674 0.164528 0.175781 0.33838 0.376069 2024-09-15T20:57:19.778 INFO:tasks.workunit.client.0.smithi139.stdout: 17 16 741 725 0.166567 0.199219 0.317103 0.372144 2024-09-15T20:57:20.778 INFO:tasks.workunit.client.0.smithi139.stdout: 18 16 789 773 0.167729 0.1875 0.326952 0.368945 2024-09-15T20:57:21.778 INFO:tasks.workunit.client.0.smithi139.stdout: 19 16 835 819 0.168357 0.179688 0.358502 0.367042 2024-09-15T20:57:22.779 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:57:22.778854+0000 min lat: 0.174764 max lat: 2.04831 avg lat: 0.365346 2024-09-15T20:57:22.779 INFO:tasks.workunit.client.0.smithi139.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T20:57:22.779 INFO:tasks.workunit.client.0.smithi139.stdout: 20 16 887 871 0.170095 0.203125 0.245724 0.365346 2024-09-15T20:57:23.779 INFO:tasks.workunit.client.0.smithi139.stdout: 21 16 930 914 0.169992 0.167969 0.237679 0.363816 2024-09-15T20:57:24.779 INFO:tasks.workunit.client.0.smithi139.stdout: 22 16 980 964 0.171142 0.195312 0.33533 0.362473 2024-09-15T20:57:25.779 INFO:tasks.workunit.client.0.smithi139.stdout: 23 16 1028 1012 0.171853 0.1875 0.408734 0.361389 2024-09-15T20:57:26.779 INFO:tasks.workunit.client.0.smithi139.stdout: 24 16 1075 1059 0.172341 0.183594 0.34216 0.359635 2024-09-15T20:57:27.779 INFO:tasks.workunit.client.0.smithi139.stdout: 25 16 1125 1109 0.173259 0.195312 0.349308 0.358153 2024-09-15T20:57:28.779 INFO:tasks.workunit.client.0.smithi139.stdout: 26 16 1163 1147 0.172303 0.148438 0.375522 0.359575 2024-09-15T20:57:29.780 INFO:tasks.workunit.client.0.smithi139.stdout: 27 16 1210 1194 0.172721 0.183594 0.293326 0.359255 2024-09-15T20:57:30.780 INFO:tasks.workunit.client.0.smithi139.stdout: 28 16 1255 1239 0.172829 0.175781 0.245955 0.358461 2024-09-15T20:57:31.780 INFO:tasks.workunit.client.0.smithi139.stdout: 29 16 1302 1286 0.173199 0.183594 0.375137 0.358956 2024-09-15T20:57:32.780 INFO:tasks.workunit.client.0.smithi139.stdout: 30 16 1348 1332 0.173415 0.179688 0.235685 0.358059 2024-09-15T20:57:33.781 INFO:tasks.workunit.client.0.smithi139.stdout:Total time run: 30.2489 2024-09-15T20:57:33.781 INFO:tasks.workunit.client.0.smithi139.stdout:Total writes made: 1348 2024-09-15T20:57:33.781 INFO:tasks.workunit.client.0.smithi139.stdout:Write size: 4096 2024-09-15T20:57:33.781 INFO:tasks.workunit.client.0.smithi139.stdout:Object size: 4096 2024-09-15T20:57:33.781 INFO:tasks.workunit.client.0.smithi139.stdout:Bandwidth (MB/sec): 0.174077 2024-09-15T20:57:33.781 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev Bandwidth: 0.0461351 2024-09-15T20:57:33.781 INFO:tasks.workunit.client.0.smithi139.stdout:Max bandwidth (MB/sec): 0.214844 2024-09-15T20:57:33.781 INFO:tasks.workunit.client.0.smithi139.stdout:Min bandwidth (MB/sec): 0.0117188 2024-09-15T20:57:33.781 INFO:tasks.workunit.client.0.smithi139.stdout:Average IOPS: 44 2024-09-15T20:57:33.781 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev IOPS: 11.8106 2024-09-15T20:57:33.781 INFO:tasks.workunit.client.0.smithi139.stdout:Max IOPS: 55 2024-09-15T20:57:33.781 INFO:tasks.workunit.client.0.smithi139.stdout:Min IOPS: 3 2024-09-15T20:57:33.782 INFO:tasks.workunit.client.0.smithi139.stdout:Average Latency(s): 0.358108 2024-09-15T20:57:33.782 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev Latency(s): 0.176968 2024-09-15T20:57:33.782 INFO:tasks.workunit.client.0.smithi139.stdout:Max latency(s): 2.04831 2024-09-15T20:57:33.782 INFO:tasks.workunit.client.0.smithi139.stdout:Min latency(s): 0.174764 2024-09-15T20:57:33.890 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 130342 2024-09-15T20:57:33.890 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2024-09-15T20:57:34.892 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 130342 2024-09-15T20:57:34.892 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (130342) - No such process 2024-09-15T20:57:34.892 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2024-09-15T20:57:35.535 INFO:tasks.workunit.client.0.smithi139.stderr:osd.0 is already down. 2024-09-15T20:57:35.551 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 131345 2024-09-15T20:57:35.551 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2024-09-15T20:57:36.553 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 131345 2024-09-15T20:57:36.553 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (131345) - No such process 2024-09-15T20:57:36.553 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2024-09-15T20:57:37.106 INFO:tasks.workunit.client.0.smithi139.stderr:osd.1 is already down. 2024-09-15T20:57:37.124 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 132365 2024-09-15T20:57:37.124 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2024-09-15T20:57:38.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 132365 2024-09-15T20:57:38.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (132365) - No such process 2024-09-15T20:57:38.125 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2024-09-15T20:57:38.587 INFO:tasks.workunit.client.0.smithi139.stderr:osd.2 is already down. 2024-09-15T20:57:38.604 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 133376 2024-09-15T20:57:38.605 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2024-09-15T20:57:39.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 133376 2024-09-15T20:57:39.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (133376) - No such process 2024-09-15T20:57:39.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2024-09-15T20:57:40.191 INFO:tasks.workunit.client.0.smithi139.stderr:osd.3 is already down. 2024-09-15T20:57:40.208 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-15T20:57:42.567 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:57:42.590 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2024-09-15T20:57:43.159 INFO:tasks.workunit.client.0.smithi139.stderr:1024+0 records in 2024-09-15T20:57:43.159 INFO:tasks.workunit.client.0.smithi139.stderr:1024+0 records out 2024-09-15T20:57:43.159 INFO:tasks.workunit.client.0.smithi139.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.567148 s, 1.9 GB/s 2024-09-15T20:57:43.159 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2024-09-15T20:57:43.184 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:57:53.603 INFO:tasks.workunit.client.0.smithi139.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-09-15T20:57:53.603 INFO:tasks.workunit.client.0.smithi139.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2024-09-15T20:57:53.609 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-15T20:57:55.979 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:57:55.999 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2024-09-15T20:57:56.305 INFO:tasks.workunit.client.0.smithi139.stderr:512+0 records in 2024-09-15T20:57:56.305 INFO:tasks.workunit.client.0.smithi139.stderr:512+0 records out 2024-09-15T20:57:56.305 INFO:tasks.workunit.client.0.smithi139.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.304672 s, 1.8 GB/s 2024-09-15T20:57:56.305 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2024-09-15T20:57:56.331 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:58:02.991 INFO:tasks.workunit.client.0.smithi139.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-09-15T20:58:02.991 INFO:tasks.workunit.client.0.smithi139.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2024-09-15T20:58:02.999 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2024-09-15T20:58:03.067 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-15T20:58:05.439 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:58:05.454 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-15T20:58:06.077 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:06.073+0000 7f6ed2f88ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:58:06.855 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:06.853+0000 7f6ed2f88ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:58:07.835 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:58:07.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2024-09-15T20:58:08.426 INFO:tasks.workunit.client.0.smithi139.stderr:1024+0 records in 2024-09-15T20:58:08.426 INFO:tasks.workunit.client.0.smithi139.stderr:1024+0 records out 2024-09-15T20:58:08.426 INFO:tasks.workunit.client.0.smithi139.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.565614 s, 1.9 GB/s 2024-09-15T20:58:08.426 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2024-09-15T20:58:08.450 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:58:09.580 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:09.577+0000 7ff29589cac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:58:10.363 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:10.361+0000 7ff29589cac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:58:16.987 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:16.985+0000 7ff29589cac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:58:18.860 INFO:tasks.workunit.client.0.smithi139.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-15T20:58:18.860 INFO:tasks.workunit.client.0.smithi139.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2024-09-15T20:58:18.860 INFO:tasks.workunit.client.0.smithi139.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2024-09-15T20:58:18.868 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2024-09-15T20:58:18.986 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-15T20:58:19.616 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:19.613+0000 7f2a30195ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:58:20.166 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:20.161+0000 7f2a30195ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:58:20.907 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:58:20.924 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-09-15T20:58:20.951 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:58:21.839 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:21.836+0000 7f1febab5ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:58:22.358 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:22.352+0000 7f1febab5ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:58:23.075 INFO:tasks.workunit.client.0.smithi139.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-15T20:58:23.085 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2024-09-15T20:58:23.206 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-15T20:58:24.587 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:58:24.600 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2024-09-15T20:58:26.699 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:58:26.723 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-09-15T20:58:27.292 INFO:tasks.workunit.client.0.smithi139.stderr:1024+0 records in 2024-09-15T20:58:27.292 INFO:tasks.workunit.client.0.smithi139.stderr:1024+0 records out 2024-09-15T20:58:27.292 INFO:tasks.workunit.client.0.smithi139.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.567502 s, 1.9 GB/s 2024-09-15T20:58:27.293 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-09-15T20:58:27.316 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:58:28.159 INFO:tasks.workunit.client.0.smithi139.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2024-09-15T20:58:28.161 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2024-09-15T20:58:28.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-15T20:58:28.755 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:28.752+0000 7f46b3addac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:58:29.567 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:29.564+0000 7f46b3addac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:58:30.531 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:58:30.550 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-09-15T20:58:38.237 INFO:tasks.workunit.client.0.smithi139.stderr:1024+0 records in 2024-09-15T20:58:38.238 INFO:tasks.workunit.client.0.smithi139.stderr:1024+0 records out 2024-09-15T20:58:38.238 INFO:tasks.workunit.client.0.smithi139.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.54753 s, 142 MB/s 2024-09-15T20:58:38.238 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-09-15T20:58:38.265 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:58:39.409 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:39.404+0000 7f254f325ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:58:40.191 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:40.188+0000 7f254f325ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:58:40.248 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:40.244+0000 7f254f325ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:58:41.439 INFO:tasks.workunit.client.0.smithi139.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-09-15T20:58:41.439 INFO:tasks.workunit.client.0.smithi139.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-09-15T20:58:41.439 INFO:tasks.workunit.client.0.smithi139.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2024-09-15T20:58:41.449 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-15T20:58:42.044 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:42.040+0000 7f8b1b8d7ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:58:42.575 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:42.568+0000 7f8b1b8d7ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:58:43.319 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:58:43.335 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2024-09-15T20:58:43.907 INFO:tasks.workunit.client.0.smithi139.stderr:1024+0 records in 2024-09-15T20:58:43.907 INFO:tasks.workunit.client.0.smithi139.stderr:1024+0 records out 2024-09-15T20:58:43.907 INFO:tasks.workunit.client.0.smithi139.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.570061 s, 1.9 GB/s 2024-09-15T20:58:43.907 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2024-09-15T20:58:43.932 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T20:58:53.811 INFO:tasks.workunit.client.0.smithi139.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2024-09-15T20:58:53.811 INFO:tasks.workunit.client.0.smithi139.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-09-15T20:58:53.811 INFO:tasks.workunit.client.0.smithi139.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2024-09-15T20:58:53.820 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-15T20:58:54.412 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:54.408+0000 7f264a86cac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:58:54.951 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:54.948+0000 7f264a86cac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:58:55.695 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T20:58:55.715 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-15T20:58:55.715 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:58:55.715 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:58:55.715 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T20:58:55.715 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:58:55.715 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-15T20:58:55.715 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T20:58:55.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:58:55.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:58:55.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:58:55.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-15T20:58:55.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-15T20:58:55.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:58:55.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:58:55.716 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T20:58:55.717 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:58:55.717 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:58:55.717 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:58:55.717 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:58:55.717 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:58:55.717 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:58:55.717 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:58:55.718 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:58:55.718 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:58:55.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T20:58:55.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T20:58:55.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:58:55.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:58:55.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:58:55.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:58:55.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:58:55.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:58:55.719 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-15T20:58:55.719 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T20:58:55.720 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T20:58:55.720 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:58:55.721 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-15T20:58:55.722 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T20:58:55.722 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:58:55.723 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:58:55.723 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:58:55.754 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:55.748+0000 7f4f9abcd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:58:55.757 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:55.752+0000 7f4f9abcd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:58:55.758 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:55.752+0000 7f4f9abcd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:58:56.153 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T20:58:56.153 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:58:56.153 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T20:58:56.153 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:58:56.153 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:58:56.153 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:56.153 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:58:56.154 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:58:56.154 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:56.154 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:58:56.607 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:57.494 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:58:57.488+0000 7f4f9abcd7c0 -1 Falling back to public interface 2024-09-15T20:58:57.609 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:57.609 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:57.609 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:58:57.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:58:57.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:57.610 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:58:58.044 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:58:59.046 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:58:59.046 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:58:59.046 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:58:59.047 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:58:59.047 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:58:59.047 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:58:59.482 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:00.483 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:00.483 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:00.483 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:59:00.484 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:59:00.484 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:00.484 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:59:00.957 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:01.958 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:01.958 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:01.958 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:59:01.959 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:59:01.959 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:01.959 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:59:02.375 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:02.833 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:02.828+0000 7f4f9abcd7c0 -1 osd.0 77 log_to_monitors true 2024-09-15T20:59:03.376 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:03.377 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:59:03.377 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:03.377 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:59:03.377 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:03.377 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T20:59:03.925 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 88 up_thru 88 down_at 78 last_clean_interval [64,77) [v2:127.0.0.1:6802/2156105123,v1:127.0.0.1:6803/2156105123] [v2:127.0.0.1:6804/2156105123,v1:127.0.0.1:6805/2156105123] exists,up 3b45a55c-3ec6-4b29-933b-e92e4449d4c3 2024-09-15T20:59:03.925 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:59:03.925 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:59:03.925 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:59:03.926 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-15T20:59:03.928 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=140508 2024-09-15T20:59:03.929 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-15T20:59:03.929 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:59:03.929 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:59:03.929 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-15T20:59:03.929 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:59:03.929 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-15T20:59:03.929 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T20:59:03.929 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:59:03.930 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:59:03.930 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:59:03.930 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-15T20:59:03.930 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-15T20:59:03.930 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:59:03.930 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:59:03.930 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T20:59:03.931 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:59:03.931 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:59:03.931 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:59:03.931 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:59:03.931 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:59:03.931 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:59:03.933 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:59:03.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:59:03.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:59:03.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T20:59:03.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T20:59:03.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:59:03.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:59:03.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:59:03.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:59:03.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:59:03.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:59:03.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-15T20:59:03.934 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.1 2024-09-15T20:59:03.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-15T20:59:03.935 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:59:03.935 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-15T20:59:03.935 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-15T20:59:03.935 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:59:03.935 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:59:03.935 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:59:03.956 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:03.952+0000 7f60271727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:03.977 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:03.972+0000 7f60271727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:03.979 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:03.976+0000 7f60271727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:04.459 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-15T20:59:04.459 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:59:04.459 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-15T20:59:04.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:59:04.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:59:04.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:04.460 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:59:04.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:59:04.461 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:04.461 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:59:04.888 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:04.988 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:04.984+0000 7f60271727c0 -1 Falling back to public interface 2024-09-15T20:59:05.889 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:05.890 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:05.890 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:59:05.890 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:59:05.890 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:05.891 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:59:06.325 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:07.326 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:07.327 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:07.327 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:59:07.327 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:59:07.327 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:07.327 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:59:07.743 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:08.383 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:08.379+0000 7f60271727c0 -1 osd.1 80 log_to_monitors true 2024-09-15T20:59:08.744 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:08.744 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:08.744 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:59:08.745 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:59:08.745 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:08.745 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-15T20:59:09.422 INFO:tasks.workunit.client.0.smithi139.stdout:osd.1 up in weight 1 up_from 91 up_thru 91 down_at 81 last_clean_interval [67,80) [v2:127.0.0.1:6810/1126732267,v1:127.0.0.1:6811/1126732267] [v2:127.0.0.1:6812/1126732267,v1:127.0.0.1:6813/1126732267] exists,up a49f13f9-33e9-4f82-b5ec-e8b428f16326 2024-09-15T20:59:09.422 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:59:09.422 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:59:09.423 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:59:09.423 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-15T20:59:09.427 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=141511 2024-09-15T20:59:09.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-15T20:59:09.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:59:09.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:59:09.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-15T20:59:09.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:59:09.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-15T20:59:09.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T20:59:09.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:59:09.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:59:09.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:59:09.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-15T20:59:09.428 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-15T20:59:09.429 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:59:09.429 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:59:09.429 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T20:59:09.429 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:59:09.429 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:59:09.429 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:59:09.430 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:59:09.430 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:59:09.430 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:59:09.430 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:59:09.430 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:59:09.430 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:59:09.430 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T20:59:09.430 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T20:59:09.431 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:59:09.431 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:59:09.431 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:59:09.431 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:59:09.431 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:59:09.431 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:59:09.431 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-15T20:59:09.431 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-15T20:59:09.432 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.2 2024-09-15T20:59:09.432 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:59:09.432 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-15T20:59:09.433 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-15T20:59:09.433 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:59:09.433 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:59:09.433 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:59:09.456 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:09.451+0000 7f0d2ec3d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:09.458 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:09.451+0000 7f0d2ec3d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:09.459 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:09.455+0000 7f0d2ec3d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:09.995 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-15T20:59:09.995 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:59:09.995 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-15T20:59:09.996 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:59:09.996 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:59:09.996 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:59:09.996 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:09.996 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:59:09.997 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:09.997 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:59:10.438 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:10.729 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:10.723+0000 7f0d2ec3d7c0 -1 Falling back to public interface 2024-09-15T20:59:11.439 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:11.439 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:11.440 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:59:11.440 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:59:11.441 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:11.441 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:59:11.883 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:12.685 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:12.679+0000 7f0d2ec3d7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:59:12.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:12.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:12.885 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:59:12.886 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:59:12.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:12.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:59:13.243 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:13.239+0000 7f0d2ec3d7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:59:13.335 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:13.986 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:13.979+0000 7f0d2ec3d7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:59:14.329 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:14.329 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:14.329 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:59:14.330 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:59:14.330 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:14.330 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:59:14.527 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:14.523+0000 7f0d2ec3d7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:59:14.786 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:15.083 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:15.079+0000 7f0d2ec3d7c0 -1 osd.2 83 log_to_monitors true 2024-09-15T20:59:15.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:15.788 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:15.788 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:59:15.789 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:59:15.789 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:15.789 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-15T20:59:16.252 INFO:tasks.workunit.client.0.smithi139.stdout:osd.2 up in weight 1 up_from 95 up_thru 95 down_at 84 last_clean_interval [71,83) [v2:127.0.0.1:6818/66725629,v1:127.0.0.1:6819/66725629] [v2:127.0.0.1:6820/66725629,v1:127.0.0.1:6821/66725629] exists,up 135419f4-fa78-4c50-a199-2ce283b93656 2024-09-15T20:59:16.252 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:59:16.252 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:59:16.252 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:59:16.253 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-15T20:59:16.254 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=142454 2024-09-15T20:59:16.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-15T20:59:16.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T20:59:16.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T20:59:16.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-15T20:59:16.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T20:59:16.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-15T20:59:16.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T20:59:16.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T20:59:16.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T20:59:16.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T20:59:16.255 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-15T20:59:16.256 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-15T20:59:16.256 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T20:59:16.256 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T20:59:16.256 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T20:59:16.257 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T20:59:16.257 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T20:59:16.257 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T20:59:16.257 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T20:59:16.258 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T20:59:16.258 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T20:59:16.258 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:59:16.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T20:59:16.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T20:59:16.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T20:59:16.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T20:59:16.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T20:59:16.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T20:59:16.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T20:59:16.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T20:59:16.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T20:59:16.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T20:59:16.259 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-15T20:59:16.259 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-15T20:59:16.259 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.3 2024-09-15T20:59:16.260 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T20:59:16.260 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-15T20:59:16.260 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-15T20:59:16.260 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T20:59:16.260 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T20:59:16.260 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T20:59:16.287 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:16.283+0000 7f43c3d2e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:16.517 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:16.511+0000 7f43c3d2e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:16.518 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:16.515+0000 7f43c3d2e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T20:59:16.863 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-15T20:59:16.863 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T20:59:16.863 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-15T20:59:16.863 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T20:59:16.863 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T20:59:16.863 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:16.864 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T20:59:16.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T20:59:16.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:16.864 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:59:17.280 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:18.281 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:18.282 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:18.282 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T20:59:18.282 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T20:59:18.282 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:18.283 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:59:18.305 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:18.299+0000 7f43c3d2e7c0 -1 Falling back to public interface 2024-09-15T20:59:18.696 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:19.698 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:19.698 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:19.698 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T20:59:19.698 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T20:59:19.699 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:19.699 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:59:20.162 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:20.469 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:20.463+0000 7f43c3d2e7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:59:21.003 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:20.995+0000 7f43c3d2e7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:59:21.164 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:21.164 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:21.164 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T20:59:21.164 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T20:59:21.165 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:21.165 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:59:21.603 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:21.804 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:21.799+0000 7f43c3d2e7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:59:22.347 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:22.343+0000 7f43c3d2e7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-15T20:59:22.605 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:22.605 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:22.605 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T20:59:22.606 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T20:59:22.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:22.606 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:59:22.706 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T20:59:22.699+0000 7f43c3d2e7c0 -1 osd.3 85 log_to_monitors true 2024-09-15T20:59:23.038 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T20:59:24.039 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T20:59:24.039 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T20:59:24.040 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T20:59:24.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T20:59:24.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T20:59:24.040 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-15T20:59:24.720 INFO:tasks.workunit.client.0.smithi139.stdout:osd.3 up in weight 1 up_from 99 up_thru 99 down_at 86 last_clean_interval [75,85) [v2:127.0.0.1:6826/2416609160,v1:127.0.0.1:6827/2416609160] [v2:127.0.0.1:6828/2416609160,v1:127.0.0.1:6829/2416609160] exists,up e342e581-4cdf-4872-af3a-c5f3021ebbfa 2024-09-15T20:59:24.720 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T20:59:24.720 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T20:59:24.720 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T20:59:24.721 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-15T20:59:24.722 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=143444 2024-09-15T20:59:24.722 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-15T20:59:24.917 INFO:tasks.workunit.client.0.smithi139.stdout:hints = 1 2024-09-15T20:59:24.917 INFO:tasks.workunit.client.0.smithi139.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-15T20:59:24.917 INFO:tasks.workunit.client.0.smithi139.stdout:Object prefix: benchmark_data_smithi139_144410 2024-09-15T20:59:24.917 INFO:tasks.workunit.client.0.smithi139.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T20:59:24.917 INFO:tasks.workunit.client.0.smithi139.stdout: 0 0 0 0 0 0 - 0 2024-09-15T20:59:25.916 INFO:tasks.workunit.client.0.smithi139.stdout: 1 16 96 80 0.312479 0.3125 0.208829 0.18032 2024-09-15T20:59:26.916 INFO:tasks.workunit.client.0.smithi139.stdout: 2 16 179 163 0.318332 0.324219 0.170558 0.185694 2024-09-15T20:59:27.916 INFO:tasks.workunit.client.0.smithi139.stdout: 3 16 257 241 0.313769 0.304688 0.12831 0.194107 2024-09-15T20:59:28.917 INFO:tasks.workunit.client.0.smithi139.stdout: 4 16 367 351 0.342731 0.429688 0.184144 0.177629 2024-09-15T20:59:29.917 INFO:tasks.workunit.client.0.smithi139.stdout: 5 16 465 449 0.350739 0.382812 0.117457 0.175234 2024-09-15T20:59:30.917 INFO:tasks.workunit.client.0.smithi139.stdout: 6 16 563 547 0.356078 0.382812 0.167761 0.173643 2024-09-15T20:59:31.917 INFO:tasks.workunit.client.0.smithi139.stdout: 7 16 653 637 0.355426 0.351562 0.270888 0.173494 2024-09-15T20:59:32.917 INFO:tasks.workunit.client.0.smithi139.stdout: 8 16 757 741 0.361774 0.40625 0.122359 0.170674 2024-09-15T20:59:33.918 INFO:tasks.workunit.client.0.smithi139.stdout: 9 16 862 846 0.367143 0.410156 0.172965 0.169268 2024-09-15T20:59:34.917 INFO:tasks.workunit.client.0.smithi139.stdout: 10 16 982 966 0.377299 0.46875 0.125406 0.16449 2024-09-15T20:59:35.918 INFO:tasks.workunit.client.0.smithi139.stdout: 11 16 1071 1055 0.374598 0.347656 0.181979 0.165142 2024-09-15T20:59:36.918 INFO:tasks.workunit.client.0.smithi139.stdout: 12 16 1171 1155 0.375928 0.390625 0.175097 0.165229 2024-09-15T20:59:37.918 INFO:tasks.workunit.client.0.smithi139.stdout: 13 16 1267 1251 0.375852 0.375 0.1845 0.165126 2024-09-15T20:59:38.918 INFO:tasks.workunit.client.0.smithi139.stdout: 14 16 1380 1364 0.380531 0.441406 0.191245 0.163739 2024-09-15T20:59:39.918 INFO:tasks.workunit.client.0.smithi139.stdout: 15 16 1476 1460 0.380159 0.375 0.174969 0.163395 2024-09-15T20:59:40.918 INFO:tasks.workunit.client.0.smithi139.stdout: 16 16 1593 1577 0.38496 0.457031 0.125855 0.161749 2024-09-15T20:59:41.918 INFO:tasks.workunit.client.0.smithi139.stdout: 17 16 1685 1669 0.383451 0.359375 0.201123 0.161412 2024-09-15T20:59:42.919 INFO:tasks.workunit.client.0.smithi139.stdout: 18 16 1789 1773 0.384714 0.40625 0.110208 0.161947 2024-09-15T20:59:43.919 INFO:tasks.workunit.client.0.smithi139.stdout: 19 16 1885 1869 0.384201 0.375 0.223799 0.162006 2024-09-15T20:59:44.919 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T20:59:44.918753+0000 min lat: 0.0909502 max lat: 0.345907 avg lat: 0.160689 2024-09-15T20:59:44.919 INFO:tasks.workunit.client.0.smithi139.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T20:59:44.919 INFO:tasks.workunit.client.0.smithi139.stdout: 20 16 1997 1981 0.386863 0.4375 0.145752 0.160689 2024-09-15T20:59:45.919 INFO:tasks.workunit.client.0.smithi139.stdout: 21 16 2093 2077 0.386296 0.375 0.228139 0.160913 2024-09-15T20:59:46.919 INFO:tasks.workunit.client.0.smithi139.stdout: 22 16 2205 2189 0.388622 0.4375 0.134039 0.160383 2024-09-15T20:59:47.920 INFO:tasks.workunit.client.0.smithi139.stdout: 23 16 2263 2247 0.381575 0.226562 0.374027 0.162673 2024-09-15T20:59:48.919 INFO:tasks.workunit.client.0.smithi139.stdout: 24 16 2375 2359 0.383903 0.4375 0.161337 0.162419 2024-09-15T20:59:49.919 INFO:tasks.workunit.client.0.smithi139.stdout: 25 16 2471 2455 0.383545 0.375 0.150753 0.162725 2024-09-15T20:59:50.919 INFO:tasks.workunit.client.0.smithi139.stdout: 26 16 2581 2565 0.385317 0.429688 0.132806 0.161806 2024-09-15T20:59:51.920 INFO:tasks.workunit.client.0.smithi139.stdout: 27 16 2677 2661 0.384934 0.375 0.174996 0.161628 2024-09-15T20:59:52.920 INFO:tasks.workunit.client.0.smithi139.stdout: 28 16 2799 2783 0.388204 0.476562 0.152692 0.160016 2024-09-15T20:59:53.920 INFO:tasks.workunit.client.0.smithi139.stdout: 29 16 2886 2870 0.386535 0.339844 0.176557 0.161028 2024-09-15T20:59:54.920 INFO:tasks.workunit.client.0.smithi139.stdout: 30 16 3004 2988 0.389014 0.460938 0.146668 0.160422 2024-09-15T20:59:55.920 INFO:tasks.workunit.client.0.smithi139.stdout:Total time run: 30.1259 2024-09-15T20:59:55.920 INFO:tasks.workunit.client.0.smithi139.stdout:Total writes made: 3004 2024-09-15T20:59:55.921 INFO:tasks.workunit.client.0.smithi139.stdout:Write size: 4096 2024-09-15T20:59:55.921 INFO:tasks.workunit.client.0.smithi139.stdout:Object size: 4096 2024-09-15T20:59:55.921 INFO:tasks.workunit.client.0.smithi139.stdout:Bandwidth (MB/sec): 0.389512 2024-09-15T20:59:55.921 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev Bandwidth: 0.0554576 2024-09-15T20:59:55.921 INFO:tasks.workunit.client.0.smithi139.stdout:Max bandwidth (MB/sec): 0.476562 2024-09-15T20:59:55.921 INFO:tasks.workunit.client.0.smithi139.stdout:Min bandwidth (MB/sec): 0.226562 2024-09-15T20:59:55.921 INFO:tasks.workunit.client.0.smithi139.stdout:Average IOPS: 99 2024-09-15T20:59:55.921 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev IOPS: 14.1971 2024-09-15T20:59:55.921 INFO:tasks.workunit.client.0.smithi139.stdout:Max IOPS: 122 2024-09-15T20:59:55.921 INFO:tasks.workunit.client.0.smithi139.stdout:Min IOPS: 58 2024-09-15T20:59:55.921 INFO:tasks.workunit.client.0.smithi139.stdout:Average Latency(s): 0.160307 2024-09-15T20:59:55.921 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev Latency(s): 0.0439667 2024-09-15T20:59:55.921 INFO:tasks.workunit.client.0.smithi139.stdout:Max latency(s): 0.504388 2024-09-15T20:59:55.921 INFO:tasks.workunit.client.0.smithi139.stdout:Min latency(s): 0.079535 2024-09-15T20:59:55.982 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2024-09-15T20:59:55.982 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T20:59:55.982 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T20:59:55.982 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T20:59:55.982 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T20:59:55.983 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T20:59:55.983 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T20:59:55.984 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T20:59:55.984 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T20:59:55.984 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T20:59:56.105 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T20:59:56.105 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T20:59:56.105 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T20:59:56.105 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T20:59:56.105 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T20:59:56.106 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T20:59:56.588 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-15T20:59:56.588 INFO:tasks.workunit.client.0.smithi139.stderr:1 2024-09-15T20:59:56.588 INFO:tasks.workunit.client.0.smithi139.stderr:2 2024-09-15T20:59:56.588 INFO:tasks.workunit.client.0.smithi139.stderr:3' 2024-09-15T20:59:56.588 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T20:59:56.588 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:59:56.589 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T20:59:56.777 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122061 2024-09-15T20:59:56.777 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122061 2024-09-15T20:59:56.777 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-377957122061' 2024-09-15T20:59:56.777 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:59:56.777 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T20:59:56.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=390842023948 2024-09-15T20:59:56.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 390842023948 2024-09-15T20:59:56.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-377957122061 1-390842023948' 2024-09-15T20:59:56.934 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:59:56.935 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T20:59:57.074 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=408021893131 2024-09-15T20:59:57.075 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 408021893131 2024-09-15T20:59:57.075 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-377957122061 1-390842023948 2-408021893131' 2024-09-15T20:59:57.075 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T20:59:57.075 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T20:59:57.226 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=425201762313 2024-09-15T20:59:57.226 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 425201762313 2024-09-15T20:59:57.226 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-377957122061 1-390842023948 2-408021893131 3-425201762313' 2024-09-15T20:59:57.226 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:59:57.227 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-377957122061 2024-09-15T20:59:57.227 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:59:57.227 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T20:59:57.228 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-377957122061 2024-09-15T20:59:57.228 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:59:57.229 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122061 2024-09-15T20:59:57.229 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 377957122061 2024-09-15T20:59:57.230 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 377957122061' 2024-09-15T20:59:57.230 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T20:59:57.651 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122061 -lt 377957122061 2024-09-15T20:59:57.652 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:59:57.652 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-390842023948 2024-09-15T20:59:57.652 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:59:57.654 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-15T20:59:57.655 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-390842023948 2024-09-15T20:59:57.655 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:59:57.656 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=390842023948 2024-09-15T20:59:57.657 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.1 seq 390842023948 2024-09-15T20:59:57.657 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 390842023948' 2024-09-15T20:59:57.657 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T20:59:58.205 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023948 -lt 390842023948 2024-09-15T20:59:58.206 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:59:58.206 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-408021893131 2024-09-15T20:59:58.207 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:59:58.208 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-15T20:59:58.210 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-408021893131 2024-09-15T20:59:58.210 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:59:58.211 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=408021893131 2024-09-15T20:59:58.211 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.2 seq 408021893131 2024-09-15T20:59:58.212 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 408021893131' 2024-09-15T20:59:58.212 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T20:59:58.693 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 408021893131 -lt 408021893131 2024-09-15T20:59:58.693 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T20:59:58.694 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-425201762313 2024-09-15T20:59:58.694 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T20:59:58.696 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-15T20:59:58.696 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-425201762313 2024-09-15T20:59:58.697 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T20:59:58.698 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=425201762313 2024-09-15T20:59:58.698 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.3 seq 425201762313 2024-09-15T20:59:58.698 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 425201762313' 2024-09-15T20:59:58.698 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T20:59:59.114 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 425201762311 -lt 425201762313 2024-09-15T20:59:59.115 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:00:00.116 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:00:00.117 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T21:00:00.528 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 425201762314 -lt 425201762313 2024-09-15T21:00:00.529 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:00:00.530 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:00:00.530 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:00:01.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-15T21:00:01.096 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:00:01.097 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:00:01.097 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:00:01.097 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:00:01.097 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:00:01.097 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:00:01.097 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:00:01.525 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-15T21:00:01.526 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:00:01.526 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:00:01.526 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:00:02.066 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-15T21:00:02.066 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:00:02.066 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:00:02.066 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-15T21:00:02.066 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-15T21:00:02.066 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:00:02.066 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-15T21:00:02.067 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:00:02.067 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:00:02.067 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:00:02.067 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:00:02.068 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:00:02.380 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:00:02.381 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:00:02.382 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:00:02.382 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:00:02.384 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:00:02.384 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:00:02.384 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:00:02.385 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:00:02.385 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:00:02.385 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:00:02.386 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:00:02.387 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:00:02.388 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:00:02.388 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-15T21:00:02.859 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:00:02.859 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:00:02.859 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T21:00:02.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.97169 2024-09-15T21:00:02.861 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:00:02.861 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:00:02.861 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-15T21:00:02.861 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-15T21:00:02.861 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bluefs-volume-ops 2024-09-15T21:00:02.861 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bluefs-volume-ops 2024-09-15T21:00:02.861 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-15T21:00:02.861 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:00:02.861 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-15T21:00:02.862 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:00:02.862 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:00:02.862 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:00:02.862 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:00:02.862 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:00:02.866 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:00:02.866 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:00:02.867 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:00:02.868 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:00:02.869 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:00:02.869 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:00:02.869 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:00:02.870 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:00:02.870 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:00:02.871 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:00:02.871 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:00:02.872 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:00:02.873 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:00:02.873 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-15T21:00:02.875 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:00:02.875 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:00:02.875 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T21:00:02.875 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.97169 2024-09-15T21:00:02.876 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:00:02.876 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:00:02.876 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-15T21:00:02.878 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:00:02.878 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:00:02.878 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T21:00:02.878 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.97169 2024-09-15T21:00:02.880 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:00:02.880 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:00:02.881 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:00:02.881 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-15T21:00:02.881 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore2 td/osd-bluefs-volume-ops 2024-09-15T21:00:02.881 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:341: TEST_bluestore2: local dir=td/osd-bluefs-volume-ops 2024-09-15T21:00:02.882 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: ulimit -n 2024-09-15T21:00:02.882 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: local flimit=4096 2024-09-15T21:00:02.882 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:344: TEST_bluestore2: '[' 4096 -lt 1536 ']' 2024-09-15T21:00:02.882 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: export CEPH_MON=127.0.0.1:7146 2024-09-15T21:00:02.882 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: CEPH_MON=127.0.0.1:7146 2024-09-15T21:00:02.882 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:348: TEST_bluestore2: export CEPH_ARGS 2024-09-15T21:00:02.882 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: uuidgen 2024-09-15T21:00:02.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: CEPH_ARGS+='--fsid=ac260b83-7717-443c-941d-4c370eee4e20 --auth-supported=none ' 2024-09-15T21:00:02.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:350: TEST_bluestore2: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-15T21:00:02.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:351: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-09-15T21:00:02.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:352: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-15T21:00:02.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:353: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-15T21:00:02.886 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:354: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-09-15T21:00:02.887 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:355: TEST_bluestore2: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-15T21:00:02.887 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:356: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-09-15T21:00:02.887 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:357: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-09-15T21:00:02.887 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:358: TEST_bluestore2: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-09-15T21:00:02.887 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:360: TEST_bluestore2: run_mon td/osd-bluefs-volume-ops a 2024-09-15T21:00:02.887 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-15T21:00:02.887 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:00:02.887 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:00:02.887 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:00:02.887 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-15T21:00:02.887 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-15T21:00:03.381 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:00:03.381 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:00:03.381 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:00:03.381 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:00:03.381 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:00:03.381 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T21:00:03.382 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T21:00:03.382 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T21:00:03.594 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:00:03.595 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:00:03.595 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:00:03.595 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:00:03.595 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:00:03.596 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:00:03.596 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:00:03.596 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:00:03.596 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:00:03.597 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:00:03.597 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:00:03.597 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T21:00:03.597 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.97169/ceph-mon.a.asok 2024-09-15T21:00:03.597 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:00:03.597 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.97169/ceph-mon.a.asok config get fsid 2024-09-15T21:00:03.711 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:00:03.712 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:00:03.712 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:00:03.712 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:00:03.712 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:00:03.712 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:00:03.712 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:00:03.712 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:00:03.713 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:00:03.713 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:00:03.713 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T21:00:03.713 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.97169/ceph-mon.a.asok 2024-09-15T21:00:03.714 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:00:03.714 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.97169/ceph-mon.a.asok config get mon_host 2024-09-15T21:00:03.815 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:361: TEST_bluestore2: run_mgr td/osd-bluefs-volume-ops x 2024-09-15T21:00:03.815 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-15T21:00:03.815 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:00:03.815 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:00:03.815 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:00:03.815 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-15T21:00:03.815 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:00:04.731 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:00:04.732 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:00:04.732 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:00:04.732 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:00:04.732 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:00:04.732 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T21:00:04.732 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T21:00:04.733 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:00:04.733 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:00:04.808 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:362: TEST_bluestore2: run_osd td/osd-bluefs-volume-ops 0 2024-09-15T21:00:04.808 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T21:00:04.808 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:00:04.808 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:00:04.808 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:00:04.808 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-15T21:00:04.808 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=ac260b83-7717-443c-941d-4c370eee4e20 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-15T21:00:04.809 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:00:04.809 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:00:04.809 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:00:04.809 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-15T21:00:04.809 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-15T21:00:04.809 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:00:04.809 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:00:04.809 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T21:00:04.809 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:00:04.809 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:00:04.809 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:00:04.809 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:00:04.809 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:00:04.809 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T21:00:04.810 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T21:00:04.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T21:00:04.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:00:04.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:00:04.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:00:04.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T21:00:04.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T21:00:04.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:00:04.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:00:04.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:00:04.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:00:04.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:00:04.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:00:04.810 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-15T21:00:04.811 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:00:04.811 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e9c02d11-4449-44cb-87f5-eca48e0c1395 2024-09-15T21:00:04.811 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e9c02d11-4449-44cb-87f5-eca48e0c1395' 2024-09-15T21:00:04.811 INFO:tasks.workunit.client.0.smithi139.stdout:add osd0 e9c02d11-4449-44cb-87f5-eca48e0c1395 2024-09-15T21:00:04.812 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:00:04.832 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDUSudmX+WBMRAAX2BjvOEfsS5ErLMjEnL6qQ== 2024-09-15T21:00:04.832 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDUSudmX+WBMRAAX2BjvOEfsS5ErLMjEnL6qQ=="}' 2024-09-15T21:00:04.832 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e9c02d11-4449-44cb-87f5-eca48e0c1395 -i td/osd-bluefs-volume-ops/0/new.json 2024-09-15T21:00:05.195 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T21:00:05.212 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-15T21:00:05.213 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=ac260b83-7717-443c-941d-4c370eee4e20 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDUSudmX+WBMRAAX2BjvOEfsS5ErLMjEnL6qQ== --osd-uuid e9c02d11-4449-44cb-87f5-eca48e0c1395 2024-09-15T21:00:05.251 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:00:05.242+0000 7f19ac7ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:00:05.272 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:00:05.266+0000 7f19ac7ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:00:05.274 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:00:05.266+0000 7f19ac7ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:00:05.275 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:00:05.270+0000 7f19ac7ed7c0 -1 bdev(0x5600c54f4800 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:00:05.275 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:00:05.270+0000 7f19ac7ed7c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-15T21:00:13.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-15T21:00:13.856 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:00:13.857 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:00:13.858 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd0 key to auth repository 2024-09-15T21:00:13.858 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:00:14.438 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T21:00:14.438 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:00:14.439 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=ac260b83-7717-443c-941d-4c370eee4e20 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:00:14.439 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:00:14.439 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:00:14.439 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:00:14.475 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:00:14.474+0000 7fb0607367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:00:14.502 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:00:14.498+0000 7fb0607367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:00:14.503 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:00:14.502+0000 7fb0607367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:00:14.896 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:00:14.896 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:00:14.896 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:00:14.896 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:00:14.896 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:00:14.896 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:14.896 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T21:00:14.897 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:00:14.897 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:14.897 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:00:15.357 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:16.225 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:00:16.222+0000 7fb0607367c0 -1 Falling back to public interface 2024-09-15T21:00:16.359 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:16.359 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:16.359 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:00:16.359 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T21:00:16.360 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:16.360 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:00:16.790 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:17.792 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:17.792 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:17.792 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T21:00:17.792 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:00:17.793 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:17.793 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:00:18.258 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:19.260 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:19.260 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:19.260 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:00:19.260 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T21:00:19.261 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:19.261 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:00:19.677 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:20.679 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:20.679 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:20.679 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T21:00:20.680 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:00:20.680 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:20.680 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:00:20.813 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:00:20.810+0000 7fb0607367c0 -1 osd.0 0 log_to_monitors true 2024-09-15T21:00:21.184 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:22.185 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:22.186 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:22.186 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T21:00:22.186 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:00:22.186 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:22.186 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:00:22.688 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:23.690 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:23.690 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:23.690 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T21:00:23.691 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:00:23.691 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:23.691 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:00:25.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:00:26.199 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:00:26.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:00:26.200 INFO:tasks.workunit.client.0.smithi139.stdout:7 2024-09-15T21:00:26.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-15T21:00:26.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:00:26.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:00:26.700 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1683854740,v1:127.0.0.1:6803/1683854740] [v2:127.0.0.1:6804/1683854740,v1:127.0.0.1:6805/1683854740] exists,up e9c02d11-4449-44cb-87f5-eca48e0c1395 2024-09-15T21:00:26.700 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:00:26.700 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:00:26.700 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:00:26.701 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-15T21:00:26.702 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: osd_pid0=146309 2024-09-15T21:00:26.702 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:365: TEST_bluestore2: sleep 5 2024-09-15T21:00:31.704 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:366: TEST_bluestore2: create_pool foo 16 2024-09-15T21:00:31.704 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2024-09-15T21:00:32.253 INFO:tasks.workunit.client.0.smithi139.stderr:pool 'foo' already exists 2024-09-15T21:00:32.269 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:00:33.271 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:368: TEST_bluestore2: retry = 0 2024-09-15T21:00:33.271 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 368: retry: command not found 2024-09-15T21:00:33.272 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ '' -le 5 ]] 2024-09-15T21:00:33.272 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2024-09-15T21:00:33.315 INFO:tasks.workunit.client.0.smithi139.stdout:hints = 1 2024-09-15T21:00:33.317 INFO:tasks.workunit.client.0.smithi139.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2024-09-15T21:00:33.317 INFO:tasks.workunit.client.0.smithi139.stdout:Object prefix: benchmark_data_smithi139_147625 2024-09-15T21:00:33.351 INFO:tasks.workunit.client.0.smithi139.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T21:00:33.351 INFO:tasks.workunit.client.0.smithi139.stdout: 0 0 0 0 0 0 - 0 2024-09-15T21:00:34.351 INFO:tasks.workunit.client.0.smithi139.stdout: 1 16 17 1 3.99942 4 0.246775 0.246775 2024-09-15T21:00:35.351 INFO:tasks.workunit.client.0.smithi139.stdout: 2 16 33 17 33.9946 64 1.29568 1.36396 2024-09-15T21:00:36.351 INFO:tasks.workunit.client.0.smithi139.stdout: 3 16 49 33 43.993 64 0.960238 1.13775 2024-09-15T21:00:37.352 INFO:tasks.workunit.client.0.smithi139.stdout: 4 16 65 49 48.9922 64 1.08579 1.14229 2024-09-15T21:00:38.352 INFO:tasks.workunit.client.0.smithi139.stdout: 5 16 81 65 51.9917 64 0.834687 1.06667 2024-09-15T21:00:39.352 INFO:tasks.workunit.client.0.smithi139.stdout: 6 16 97 81 53.9914 64 1.49571 1.13924 2024-09-15T21:00:40.352 INFO:tasks.workunit.client.0.smithi139.stdout: 7 16 97 81 46.2783 0 - 1.13924 2024-09-15T21:00:41.352 INFO:tasks.workunit.client.0.smithi139.stdout: 8 16 113 97 48.4923 32 1.47852 1.19442 2024-09-15T21:00:42.352 INFO:tasks.workunit.client.0.smithi139.stdout: 9 16 129 113 50.2142 64 0.9182 1.14868 2024-09-15T21:00:43.353 INFO:tasks.workunit.client.0.smithi139.stdout: 10 16 145 129 51.5917 64 1.26365 1.1594 2024-09-15T21:00:44.353 INFO:tasks.workunit.client.0.smithi139.stdout:Total time run: 10.5145 2024-09-15T21:00:44.353 INFO:tasks.workunit.client.0.smithi139.stdout:Total writes made: 145 2024-09-15T21:00:44.353 INFO:tasks.workunit.client.0.smithi139.stdout:Write size: 4194304 2024-09-15T21:00:44.353 INFO:tasks.workunit.client.0.smithi139.stdout:Object size: 4194304 2024-09-15T21:00:44.353 INFO:tasks.workunit.client.0.smithi139.stdout:Bandwidth (MB/sec): 55.1617 2024-09-15T21:00:44.353 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev Bandwidth: 26.4289 2024-09-15T21:00:44.353 INFO:tasks.workunit.client.0.smithi139.stdout:Max bandwidth (MB/sec): 64 2024-09-15T21:00:44.354 INFO:tasks.workunit.client.0.smithi139.stdout:Min bandwidth (MB/sec): 0 2024-09-15T21:00:44.354 INFO:tasks.workunit.client.0.smithi139.stdout:Average IOPS: 13 2024-09-15T21:00:44.354 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev IOPS: 6.60724 2024-09-15T21:00:44.354 INFO:tasks.workunit.client.0.smithi139.stdout:Max IOPS: 16 2024-09-15T21:00:44.354 INFO:tasks.workunit.client.0.smithi139.stdout:Min IOPS: 0 2024-09-15T21:00:44.354 INFO:tasks.workunit.client.0.smithi139.stdout:Average Latency(s): 1.14295 2024-09-15T21:00:44.354 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev Latency(s): 0.255608 2024-09-15T21:00:44.354 INFO:tasks.workunit.client.0.smithi139.stdout:Max latency(s): 1.49571 2024-09-15T21:00:44.354 INFO:tasks.workunit.client.0.smithi139.stdout:Min latency(s): 0.246775 2024-09-15T21:00:44.413 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2024-09-15T21:00:54.417 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-15T21:00:54.417 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2024-09-15T21:00:54.595 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=1038090240 2024-09-15T21:00:54.595 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-15T21:00:54.595 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2024-09-15T21:00:54.768 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=0 2024-09-15T21:00:54.768 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2024-09-15T21:00:54.769 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 0 -eq 0 2024-09-15T21:00:54.769 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ 1 -le 5 ]] 2024-09-15T21:00:54.769 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2024-09-15T21:00:54.802 INFO:tasks.workunit.client.0.smithi139.stdout:hints = 1 2024-09-15T21:00:54.804 INFO:tasks.workunit.client.0.smithi139.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2024-09-15T21:00:54.804 INFO:tasks.workunit.client.0.smithi139.stdout:Object prefix: benchmark_data_smithi139_147698 2024-09-15T21:00:54.835 INFO:tasks.workunit.client.0.smithi139.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T21:00:54.835 INFO:tasks.workunit.client.0.smithi139.stdout: 0 0 0 0 0 0 - 0 2024-09-15T21:00:55.835 INFO:tasks.workunit.client.0.smithi139.stdout: 1 16 33 17 67.9926 68 0.759122 0.669404 2024-09-15T21:00:56.835 INFO:tasks.workunit.client.0.smithi139.stdout: 2 16 49 33 65.9913 64 1.0851 0.911181 2024-09-15T21:00:57.835 INFO:tasks.workunit.client.0.smithi139.stdout: 3 16 65 49 65.3244 64 0.675596 0.846969 2024-09-15T21:00:58.835 INFO:tasks.workunit.client.0.smithi139.stdout: 4 16 81 65 64.9906 64 1.21891 0.91908 2024-09-15T21:00:59.836 INFO:tasks.workunit.client.0.smithi139.stdout: 5 16 97 81 64.7903 64 1.0355 0.925242 2024-09-15T21:01:00.836 INFO:tasks.workunit.client.0.smithi139.stdout: 6 16 112 96 63.9908 60 1.30652 0.987384 2024-09-15T21:01:01.836 INFO:tasks.workunit.client.0.smithi139.stdout: 7 16 128 112 63.9907 64 0.870732 0.973989 2024-09-15T21:01:02.836 INFO:tasks.workunit.client.0.smithi139.stdout: 8 16 129 113 56.4917 4 1.06085 0.974758 2024-09-15T21:01:03.836 INFO:tasks.workunit.client.0.smithi139.stdout: 9 16 129 113 50.2147 0 - 0.974758 2024-09-15T21:01:04.836 INFO:tasks.workunit.client.0.smithi139.stdout: 10 16 129 113 45.1931 0 - 0.974758 2024-09-15T21:01:05.837 INFO:tasks.workunit.client.0.smithi139.stdout:Total time run: 10.7604 2024-09-15T21:01:05.837 INFO:tasks.workunit.client.0.smithi139.stdout:Total writes made: 129 2024-09-15T21:01:05.837 INFO:tasks.workunit.client.0.smithi139.stdout:Write size: 4194304 2024-09-15T21:01:05.837 INFO:tasks.workunit.client.0.smithi139.stdout:Object size: 4194304 2024-09-15T21:01:05.837 INFO:tasks.workunit.client.0.smithi139.stdout:Bandwidth (MB/sec): 47.9534 2024-09-15T21:01:05.837 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev Bandwidth: 30.3491 2024-09-15T21:01:05.837 INFO:tasks.workunit.client.0.smithi139.stdout:Max bandwidth (MB/sec): 68 2024-09-15T21:01:05.837 INFO:tasks.workunit.client.0.smithi139.stdout:Min bandwidth (MB/sec): 0 2024-09-15T21:01:05.838 INFO:tasks.workunit.client.0.smithi139.stdout:Average IOPS: 11 2024-09-15T21:01:05.838 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev IOPS: 7.58727 2024-09-15T21:01:05.838 INFO:tasks.workunit.client.0.smithi139.stdout:Max IOPS: 17 2024-09-15T21:01:05.838 INFO:tasks.workunit.client.0.smithi139.stdout:Min IOPS: 0 2024-09-15T21:01:05.838 INFO:tasks.workunit.client.0.smithi139.stdout:Average Latency(s): 1.31912 2024-09-15T21:01:05.838 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev Latency(s): 0.944406 2024-09-15T21:01:05.838 INFO:tasks.workunit.client.0.smithi139.stdout:Max latency(s): 3.77149 2024-09-15T21:01:05.838 INFO:tasks.workunit.client.0.smithi139.stdout:Min latency(s): 0.134535 2024-09-15T21:01:05.903 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2024-09-15T21:01:15.907 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-15T21:01:15.907 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2024-09-15T21:01:16.083 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=127926272 2024-09-15T21:01:16.084 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-15T21:01:16.084 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2024-09-15T21:01:16.248 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=369623040 2024-09-15T21:01:16.248 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2024-09-15T21:01:16.248 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 369623040 -eq 0 2024-09-15T21:01:16.249 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: break 2024-09-15T21:01:16.249 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:381: TEST_bluestore2: test 369623040 -gt 0 2024-09-15T21:01:16.249 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 146309 2024-09-15T21:01:16.249 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: sleep 1 2024-09-15T21:01:17.250 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 146309 2024-09-15T21:01:17.250 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 383: kill: (146309) - No such process 2024-09-15T21:01:17.250 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:384: TEST_bluestore2: ceph osd down 0 2024-09-15T21:01:17.709 INFO:tasks.workunit.client.0.smithi139.stderr:osd.0 is already down. 2024-09-15T21:01:17.725 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:386: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/block --command bluefs-bdev-migrate 2024-09-15T21:01:17.753 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T21:01:21.387 INFO:tasks.workunit.client.0.smithi139.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-09-15T21:01:21.404 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:391: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command bluefs-bdev-sizes 2024-09-15T21:01:21.434 INFO:tasks.workunit.client.0.smithi139.stdout:inferring bluefs devices from bluestore path 2024-09-15T21:01:22.756 INFO:tasks.workunit.client.0.smithi139.stdout:1 : device size 0xffffe000 : using 0x1dc98000(477 MiB) 2024-09-15T21:01:22.764 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:394: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command fsck 2024-09-15T21:01:25.143 INFO:tasks.workunit.client.0.smithi139.stdout:fsck success 2024-09-15T21:01:25.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:397: TEST_bluestore2: activate_osd td/osd-bluefs-volume-ops 0 2024-09-15T21:01:25.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T21:01:25.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-15T21:01:25.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-15T21:01:25.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-15T21:01:25.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-15T21:01:25.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=ac260b83-7717-443c-941d-4c370eee4e20 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-15T21:01:25.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:01:25.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:01:25.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:01:25.195 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-15T21:01:25.196 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-15T21:01:25.196 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-15T21:01:25.196 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-15T21:01:25.196 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T21:01:25.196 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-15T21:01:25.196 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:01:25.196 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:01:25.197 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:01:25.197 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:01:25.198 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T21:01:25.198 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T21:01:25.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T21:01:25.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:01:25.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T21:01:25.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T21:01:25.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:01:25.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:01:25.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:01:25.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:01:25.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-15T21:01:25.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-15T21:01:25.198 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-15T21:01:25.199 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T21:01:25.199 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-15T21:01:25.199 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=ac260b83-7717-443c-941d-4c370eee4e20 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:01:25.200 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-15T21:01:25.200 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-15T21:01:25.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-15T21:01:25.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-15T21:01:25.201 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-15T21:01:25.235 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:01:25.233+0000 7fe7226f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:25.237 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:01:25.233+0000 7fe7226f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:25.238 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:01:25.233+0000 7fe7226f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:25.754 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-15T21:01:25.755 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:01:25.755 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:01:25.755 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:01:25.755 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:01:25.755 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:25.756 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T21:01:25.756 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:01:25.757 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:25.757 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:01:26.224 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:26.461 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:01:26.457+0000 7fe7226f77c0 -1 Falling back to public interface 2024-09-15T21:01:27.225 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:27.225 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:27.226 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T21:01:27.226 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:01:27.226 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:27.226 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:01:27.700 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:28.702 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:28.703 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:28.703 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T21:01:28.703 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:01:28.703 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:28.704 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:01:29.137 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:30.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:30.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:30.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:01:30.139 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T21:01:30.139 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:30.140 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:01:30.579 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:31.297 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:01:31.293+0000 7fe7226f77c0 -1 osd.0 10 log_to_monitors true 2024-09-15T21:01:31.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:31.581 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:31.581 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T21:01:31.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:01:31.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:31.582 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:01:32.721 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:33.725 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:33.726 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:33.726 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:01:33.726 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T21:01:33.726 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:33.726 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:01:34.857 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:35.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:35.859 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:35.859 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T21:01:35.860 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:01:35.860 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:35.860 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:01:36.318 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/980231511,v1:127.0.0.1:6803/980231511] [v2:127.0.0.1:6804/980231511,v1:127.0.0.1:6805/980231511] exists,up e9c02d11-4449-44cb-87f5-eca48e0c1395 2024-09-15T21:01:36.318 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:01:36.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:01:36.319 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:01:36.319 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-15T21:01:36.320 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: osd_pid0=148929 2024-09-15T21:01:36.320 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:400: TEST_bluestore2: wait_for_clean 2024-09-15T21:01:36.320 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-15T21:01:36.320 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-15T21:01:36.321 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-15T21:01:36.322 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T21:01:36.322 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-15T21:01:36.322 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-15T21:01:36.322 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-15T21:01:36.322 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-15T21:01:36.322 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-15T21:01:36.432 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T21:01:36.432 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-15T21:01:36.432 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-15T21:01:36.432 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-15T21:01:36.432 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-15T21:01:36.433 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-15T21:01:36.872 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-15T21:01:36.872 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-15T21:01:36.872 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-15T21:01:36.872 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T21:01:37.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574852 2024-09-15T21:01:37.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574852 2024-09-15T21:01:37.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-55834574852' 2024-09-15T21:01:37.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-15T21:01:37.035 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-55834574852 2024-09-15T21:01:37.035 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-15T21:01:37.036 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-15T21:01:37.037 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-55834574852 2024-09-15T21:01:37.037 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-15T21:01:37.038 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574852 2024-09-15T21:01:37.038 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2024-09-15T21:01:37.039 INFO:tasks.workunit.client.0.smithi139.stdout:waiting osd.0 seq 55834574852 2024-09-15T21:01:37.039 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:01:37.448 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 55834574852 2024-09-15T21:01:37.449 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-15T21:01:38.450 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T21:01:38.450 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T21:01:38.933 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574852 -lt 55834574852 2024-09-15T21:01:38.933 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-15T21:01:38.934 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:01:38.934 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:01:39.563 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-15T21:01:39.563 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-15T21:01:39.564 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-15T21:01:39.564 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-15T21:01:39.565 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T21:01:39.565 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T21:01:39.565 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T21:01:39.565 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T21:01:39.991 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-15T21:01:39.992 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T21:01:39.992 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-15T21:01:39.992 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T21:01:40.531 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-15T21:01:40.531 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-15T21:01:40.531 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-15T21:01:40.531 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-15T21:01:40.531 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-15T21:01:40.532 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:01:40.532 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-15T21:01:40.532 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:01:40.532 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:01:40.532 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:01:40.532 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:01:40.532 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:01:40.639 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:01:40.639 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:01:40.640 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:01:40.640 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:01:40.641 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:01:40.641 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:01:40.642 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:01:40.642 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:01:40.642 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:01:40.643 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:01:40.643 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:01:40.643 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:01:40.644 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:01:40.644 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-15T21:01:40.675 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:01:40.675 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:01:40.675 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T21:01:40.675 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.97169 2024-09-15T21:01:40.677 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:01:40.677 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:01:40.677 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-15T21:01:40.677 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-15T21:01:40.677 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bluefs-volume-ops 2024-09-15T21:01:40.677 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bluefs-volume-ops 2024-09-15T21:01:40.677 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-15T21:01:40.677 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-15T21:01:40.677 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-15T21:01:40.677 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:01:40.678 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:01:40.678 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:01:40.678 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:01:40.678 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:01:40.679 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:01:40.679 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:01:40.680 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:01:40.680 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:01:40.681 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:01:40.681 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:01:40.681 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:01:40.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:01:40.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:01:40.683 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:01:40.683 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:01:40.684 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:01:40.684 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T21:01:40.684 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-15T21:01:40.686 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:01:40.686 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:01:40.686 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T21:01:40.686 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.97169 2024-09-15T21:01:40.686 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:01:40.686 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:01:40.687 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-15T21:01:40.688 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-15T21:01:40.688 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:01:40.688 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T21:01:40.688 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.97169 2024-09-15T21:01:40.689 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-15T21:01:40.695 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-15T21:01:40.695 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-15T21:01:40.695 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-15T21:01:40.695 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore_expand td/osd-bluefs-volume-ops 2024-09-15T21:01:40.695 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:404: TEST_bluestore_expand: local dir=td/osd-bluefs-volume-ops 2024-09-15T21:01:40.695 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: ulimit -n 2024-09-15T21:01:40.695 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: local flimit=4096 2024-09-15T21:01:40.695 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:407: TEST_bluestore_expand: '[' 4096 -lt 1536 ']' 2024-09-15T21:01:40.695 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: export CEPH_MON=127.0.0.1:7146 2024-09-15T21:01:40.696 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: CEPH_MON=127.0.0.1:7146 2024-09-15T21:01:40.696 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:411: TEST_bluestore_expand: export CEPH_ARGS 2024-09-15T21:01:40.696 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: uuidgen 2024-09-15T21:01:40.696 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: CEPH_ARGS+='--fsid=5f57d922-cde4-4eb0-aadb-690f156b2ba3 --auth-supported=none ' 2024-09-15T21:01:40.696 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:413: TEST_bluestore_expand: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-15T21:01:40.696 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:414: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-09-15T21:01:40.696 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:415: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-15T21:01:40.696 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:416: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-15T21:01:40.696 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:417: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-09-15T21:01:40.696 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:418: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-15T21:01:40.697 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:419: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-09-15T21:01:40.697 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:420: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-09-15T21:01:40.697 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:421: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-09-15T21:01:40.697 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:423: TEST_bluestore_expand: run_mon td/osd-bluefs-volume-ops a 2024-09-15T21:01:40.697 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-15T21:01:40.697 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-15T21:01:40.697 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-15T21:01:40.697 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-15T21:01:40.697 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-15T21:01:40.698 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-15T21:01:41.167 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-15T21:01:41.167 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:01:41.167 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:01:41.168 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:01:41.168 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:01:41.168 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T21:01:41.168 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T21:01:41.168 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T21:01:41.384 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-15T21:01:41.385 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-15T21:01:41.385 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:01:41.386 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:01:41.386 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-15T21:01:41.386 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-15T21:01:41.386 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:01:41.386 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:01:41.386 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:01:41.387 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:01:41.387 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:01:41.387 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T21:01:41.387 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.97169/ceph-mon.a.asok 2024-09-15T21:01:41.388 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:01:41.388 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.97169/ceph-mon.a.asok config get fsid 2024-09-15T21:01:41.497 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-15T21:01:41.499 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-15T21:01:41.500 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-15T21:01:41.500 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-15T21:01:41.500 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-15T21:01:41.501 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-15T21:01:41.501 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-15T21:01:41.501 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-15T21:01:41.501 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-15T21:01:41.501 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:01:41.502 INFO:tasks.workunit.client.0.smithi139.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T21:01:41.502 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.97169/ceph-mon.a.asok 2024-09-15T21:01:41.502 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-15T21:01:41.502 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.97169/ceph-mon.a.asok config get mon_host 2024-09-15T21:01:41.600 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:424: TEST_bluestore_expand: run_mgr td/osd-bluefs-volume-ops x 2024-09-15T21:01:41.600 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-15T21:01:41.600 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-15T21:01:41.600 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-15T21:01:41.600 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-15T21:01:41.601 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-15T21:01:41.601 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T21:01:42.516 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-15T21:01:42.516 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:01:42.517 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:01:42.517 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:01:42.517 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:01:42.517 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T21:01:42.517 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T21:01:42.518 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:01:42.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T21:01:42.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:425: TEST_bluestore_expand: run_osd td/osd-bluefs-volume-ops 0 2024-09-15T21:01:42.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T21:01:42.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-15T21:01:42.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-15T21:01:42.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-15T21:01:42.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-15T21:01:42.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=ac260b83-7717-443c-941d-4c370eee4e20 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=5f57d922-cde4-4eb0-aadb-690f156b2ba3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-15T21:01:42.583 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T21:01:42.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T21:01:42.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T21:01:42.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-15T21:01:42.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-15T21:01:42.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-15T21:01:42.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-15T21:01:42.584 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T21:01:42.588 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-15T21:01:42.588 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-15T21:01:42.588 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-15T21:01:42.588 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-15T21:01:42.588 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:01:42.588 INFO:tasks.workunit.client.0.smithi139.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T21:01:42.588 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T21:01:42.589 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' 2024-09-15T21:01:42.589 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T21:01:42.589 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T21:01:42.589 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T21:01:42.589 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T21:01:42.589 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T21:01:42.590 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T21:01:42.590 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T21:01:42.590 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T21:01:42.590 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T21:01:42.590 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-15T21:01:42.590 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-15T21:01:42.591 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-15T21:01:42.591 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-15T21:01:42.591 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c942dc33-466a-407a-93d2-bc803571b2a4 2024-09-15T21:01:42.591 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 c942dc33-466a-407a-93d2-bc803571b2a4' 2024-09-15T21:01:42.591 INFO:tasks.workunit.client.0.smithi139.stdout:add osd0 c942dc33-466a-407a-93d2-bc803571b2a4 2024-09-15T21:01:42.592 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-15T21:01:42.618 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA2S+dmBJ3HJBAA1FfYCpKnGG2UWM2cb05iBA== 2024-09-15T21:01:42.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA2S+dmBJ3HJBAA1FfYCpKnGG2UWM2cb05iBA=="}' 2024-09-15T21:01:42.619 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c942dc33-466a-407a-93d2-bc803571b2a4 -i td/osd-bluefs-volume-ops/0/new.json 2024-09-15T21:01:42.923 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T21:01:42.939 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-15T21:01:42.940 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=ac260b83-7717-443c-941d-4c370eee4e20 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=5f57d922-cde4-4eb0-aadb-690f156b2ba3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA2S+dmBJ3HJBAA1FfYCpKnGG2UWM2cb05iBA== --osd-uuid c942dc33-466a-407a-93d2-bc803571b2a4 2024-09-15T21:01:42.978 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:01:42.972+0000 7ff603ef87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:42.991 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:01:42.988+0000 7ff603ef87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:42.993 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:01:42.988+0000 7ff603ef87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:42.994 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:01:42.988+0000 7ff603ef87c0 -1 bdev(0x5630a8cee800 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-09-15T21:01:42.994 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:01:42.988+0000 7ff603ef87c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-15T21:01:50.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-15T21:01:50.564 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-15T21:01:50.566 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-15T21:01:50.566 INFO:tasks.workunit.client.0.smithi139.stdout:adding osd0 key to auth repository 2024-09-15T21:01:50.566 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T21:01:51.154 INFO:tasks.workunit.client.0.smithi139.stdout:start osd.0 2024-09-15T21:01:51.155 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-15T21:01:51.155 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=fe71e119-b959-4ed9-96d9-f3ab44716651 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=ac260b83-7717-443c-941d-4c370eee4e20 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=5f57d922-cde4-4eb0-aadb-690f156b2ba3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97169/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T21:01:51.155 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-15T21:01:51.155 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-15T21:01:51.155 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-15T21:01:51.193 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:01:51.188+0000 7fe7bfae17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:51.212 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:01:51.208+0000 7fe7bfae17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:51.213 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:01:51.208+0000 7fe7bfae17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T21:01:51.591 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-15T21:01:51.592 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-15T21:01:51.592 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-15T21:01:51.592 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-15T21:01:51.592 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-15T21:01:51.592 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:51.592 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-15T21:01:51.592 INFO:tasks.workunit.client.0.smithi139.stdout:0 2024-09-15T21:01:51.592 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:51.593 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:01:52.023 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:52.460 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:01:52.452+0000 7fe7bfae17c0 -1 Falling back to public interface 2024-09-15T21:01:53.025 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:53.025 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:53.025 INFO:tasks.workunit.client.0.smithi139.stdout:1 2024-09-15T21:01:53.026 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-15T21:01:53.026 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:53.026 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:01:53.457 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:54.459 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:54.459 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:54.459 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-15T21:01:54.460 INFO:tasks.workunit.client.0.smithi139.stdout:2 2024-09-15T21:01:54.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:54.460 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:01:54.946 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:55.947 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:55.947 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:55.947 INFO:tasks.workunit.client.0.smithi139.stdout:3 2024-09-15T21:01:55.948 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-15T21:01:55.948 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:55.948 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:01:56.450 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:56.992 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:01:56.988+0000 7fe7bfae17c0 -1 osd.0 0 log_to_monitors true 2024-09-15T21:01:57.451 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:57.452 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:57.452 INFO:tasks.workunit.client.0.smithi139.stdout:4 2024-09-15T21:01:57.453 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-15T21:01:57.453 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:57.453 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:01:58.516 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:01:59.517 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:01:59.518 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:01:59.518 INFO:tasks.workunit.client.0.smithi139.stdout:5 2024-09-15T21:01:59.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-15T21:01:59.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:01:59.519 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:02:01.118 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-15T21:02:02.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-15T21:02:02.120 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-15T21:02:02.120 INFO:tasks.workunit.client.0.smithi139.stdout:6 2024-09-15T21:02:02.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-15T21:02:02.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-15T21:02:02.121 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-15T21:02:02.652 INFO:tasks.workunit.client.0.smithi139.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3185031173,v1:127.0.0.1:6803/3185031173] [v2:127.0.0.1:6804/3185031173,v1:127.0.0.1:6805/3185031173] exists,up c942dc33-466a-407a-93d2-bc803571b2a4 2024-09-15T21:02:02.652 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-15T21:02:02.652 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-15T21:02:02.652 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-15T21:02:02.653 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-15T21:02:02.654 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: osd_pid0=151635 2024-09-15T21:02:02.654 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:428: TEST_bluestore_expand: sleep 5 2024-09-15T21:02:07.656 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:429: TEST_bluestore_expand: create_pool foo 16 2024-09-15T21:02:07.656 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2024-09-15T21:02:08.215 INFO:tasks.workunit.client.0.smithi139.stderr:pool 'foo' already exists 2024-09-15T21:02:08.230 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-15T21:02:09.232 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:432: TEST_bluestore_expand: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-15T21:02:09.303 INFO:tasks.workunit.client.0.smithi139.stdout:hints = 1 2024-09-15T21:02:09.303 INFO:tasks.workunit.client.0.smithi139.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-15T21:02:09.303 INFO:tasks.workunit.client.0.smithi139.stdout:Object prefix: benchmark_data_smithi139_152670 2024-09-15T21:02:09.303 INFO:tasks.workunit.client.0.smithi139.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T21:02:09.304 INFO:tasks.workunit.client.0.smithi139.stdout: 0 0 0 0 0 0 - 0 2024-09-15T21:02:10.303 INFO:tasks.workunit.client.0.smithi139.stdout: 1 16 128 112 0.43744 0.4375 0.0997299 0.13988 2024-09-15T21:02:11.303 INFO:tasks.workunit.client.0.smithi139.stdout: 2 16 273 257 0.501881 0.566406 0.100423 0.11866 2024-09-15T21:02:12.304 INFO:tasks.workunit.client.0.smithi139.stdout: 3 16 401 385 0.501229 0.5 0.09194 0.121631 2024-09-15T21:02:13.304 INFO:tasks.workunit.client.0.smithi139.stdout: 4 16 560 544 0.531173 0.621094 0.101268 0.115938 2024-09-15T21:02:14.304 INFO:tasks.workunit.client.0.smithi139.stdout: 5 16 705 689 0.538203 0.566406 0.0921711 0.115213 2024-09-15T21:02:15.304 INFO:tasks.workunit.client.0.smithi139.stdout: 6 16 865 849 0.552654 0.625 0.121785 0.111207 2024-09-15T21:02:16.304 INFO:tasks.workunit.client.0.smithi139.stdout: 7 16 1009 993 0.554048 0.5625 0.100353 0.111509 2024-09-15T21:02:17.304 INFO:tasks.workunit.client.0.smithi139.stdout: 8 16 1136 1120 0.546794 0.496094 0.0928164 0.113372 2024-09-15T21:02:18.305 INFO:tasks.workunit.client.0.smithi139.stdout: 9 16 1280 1264 0.54853 0.5625 0.0923918 0.113589 2024-09-15T21:02:19.305 INFO:tasks.workunit.client.0.smithi139.stdout: 10 16 1440 1424 0.55617 0.625 0.117154 0.111738 2024-09-15T21:02:20.305 INFO:tasks.workunit.client.0.smithi139.stdout: 11 16 1584 1568 0.556741 0.5625 0.0928989 0.112155 2024-09-15T21:02:21.305 INFO:tasks.workunit.client.0.smithi139.stdout: 12 16 1728 1712 0.557213 0.5625 0.118344 0.111486 2024-09-15T21:02:22.305 INFO:tasks.workunit.client.0.smithi139.stdout: 13 16 1872 1856 0.557612 0.5625 0.0934335 0.111786 2024-09-15T21:02:23.305 INFO:tasks.workunit.client.0.smithi139.stdout: 14 16 2017 2001 0.558234 0.566406 0.117176 0.111436 2024-09-15T21:02:24.305 INFO:tasks.workunit.client.0.smithi139.stdout: 15 16 2161 2145 0.558513 0.5625 0.116959 0.111382 2024-09-15T21:02:25.306 INFO:tasks.workunit.client.0.smithi139.stdout: 16 16 2305 2289 0.558756 0.5625 0.10047 0.110988 2024-09-15T21:02:26.306 INFO:tasks.workunit.client.0.smithi139.stdout: 17 16 2448 2432 0.558741 0.558594 0.100274 0.111582 2024-09-15T21:02:27.306 INFO:tasks.workunit.client.0.smithi139.stdout: 18 16 2592 2576 0.558946 0.5625 0.0922131 0.11122 2024-09-15T21:02:28.306 INFO:tasks.workunit.client.0.smithi139.stdout: 19 16 2736 2720 0.559128 0.5625 0.0834305 0.111685 2024-09-15T21:02:29.306 INFO:tasks.workunit.client.0.smithi139.stdout:2024-09-15T21:02:29.306273+0000 min lat: 0.0775032 max lat: 0.30881 avg lat: 0.111116 2024-09-15T21:02:29.306 INFO:tasks.workunit.client.0.smithi139.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T21:02:29.306 INFO:tasks.workunit.client.0.smithi139.stdout: 20 16 2881 2865 0.559488 0.566406 0.108818 0.111116 2024-09-15T21:02:30.306 INFO:tasks.workunit.client.0.smithi139.stdout: 21 16 3024 3008 0.559441 0.558594 0.101102 0.111456 2024-09-15T21:02:31.306 INFO:tasks.workunit.client.0.smithi139.stdout: 22 16 3185 3169 0.562593 0.628906 0.100463 0.11081 2024-09-15T21:02:32.307 INFO:tasks.workunit.client.0.smithi139.stdout: 23 16 3328 3312 0.562416 0.558594 0.117492 0.110973 2024-09-15T21:02:33.307 INFO:tasks.workunit.client.0.smithi139.stdout: 24 16 3473 3457 0.56258 0.566406 0.119987 0.110726 2024-09-15T21:02:34.307 INFO:tasks.workunit.client.0.smithi139.stdout: 25 16 3616 3600 0.562418 0.558594 0.117215 0.111095 2024-09-15T21:02:35.307 INFO:tasks.workunit.client.0.smithi139.stdout: 26 16 3776 3760 0.564823 0.625 0.101963 0.110541 2024-09-15T21:02:36.307 INFO:tasks.workunit.client.0.smithi139.stdout: 27 16 3904 3888 0.562419 0.5 0.117315 0.110962 2024-09-15T21:02:37.307 INFO:tasks.workunit.client.0.smithi139.stdout: 28 16 4049 4033 0.562559 0.566406 0.100391 0.110658 2024-09-15T21:02:38.307 INFO:tasks.workunit.client.0.smithi139.stdout: 29 16 4193 4177 0.562554 0.5625 0.108768 0.110755 2024-09-15T21:02:39.307 INFO:tasks.workunit.client.0.smithi139.stdout: 30 16 4353 4337 0.564634 0.625 0.10893 0.110224 2024-09-15T21:02:40.308 INFO:tasks.workunit.client.0.smithi139.stdout:Total time run: 30.0677 2024-09-15T21:02:40.308 INFO:tasks.workunit.client.0.smithi139.stdout:Total writes made: 4353 2024-09-15T21:02:40.308 INFO:tasks.workunit.client.0.smithi139.stdout:Write size: 4096 2024-09-15T21:02:40.308 INFO:tasks.workunit.client.0.smithi139.stdout:Object size: 4096 2024-09-15T21:02:40.308 INFO:tasks.workunit.client.0.smithi139.stdout:Bandwidth (MB/sec): 0.56552 2024-09-15T21:02:40.308 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev Bandwidth: 0.0420676 2024-09-15T21:02:40.308 INFO:tasks.workunit.client.0.smithi139.stdout:Max bandwidth (MB/sec): 0.628906 2024-09-15T21:02:40.308 INFO:tasks.workunit.client.0.smithi139.stdout:Min bandwidth (MB/sec): 0.4375 2024-09-15T21:02:40.308 INFO:tasks.workunit.client.0.smithi139.stdout:Average IOPS: 144 2024-09-15T21:02:40.309 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev IOPS: 10.7693 2024-09-15T21:02:40.309 INFO:tasks.workunit.client.0.smithi139.stdout:Max IOPS: 161 2024-09-15T21:02:40.309 INFO:tasks.workunit.client.0.smithi139.stdout:Min IOPS: 112 2024-09-15T21:02:40.309 INFO:tasks.workunit.client.0.smithi139.stdout:Average Latency(s): 0.110313 2024-09-15T21:02:40.309 INFO:tasks.workunit.client.0.smithi139.stdout:Stddev Latency(s): 0.0267438 2024-09-15T21:02:40.309 INFO:tasks.workunit.client.0.smithi139.stdout:Max latency(s): 0.30881 2024-09-15T21:02:40.309 INFO:tasks.workunit.client.0.smithi139.stdout:Min latency(s): 0.0744101 2024-09-15T21:02:40.359 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:433: TEST_bluestore_expand: sleep 5 2024-09-15T21:02:45.363 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-09-15T21:02:45.363 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-09-15T21:02:45.553 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: total_space_before=4294967296 2024-09-15T21:02:45.555 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-09-15T21:02:45.555 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep free 2024-09-15T21:02:45.555 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d: -f 2 2024-09-15T21:02:45.555 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-09-15T21:02:45.555 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d, -f 1 2024-09-15T21:02:45.556 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut '-d ' -f 2 2024-09-15T21:02:45.722 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: free_space_before=4277030912 2024-09-15T21:02:45.722 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 151635 2024-09-15T21:02:45.722 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: sleep 1 2024-09-15T21:02:46.723 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 151635 2024-09-15T21:02:46.723 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 439: kill: (151635) - No such process 2024-09-15T21:02:46.723 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:440: TEST_bluestore_expand: ceph osd down 0 2024-09-15T21:02:47.246 INFO:tasks.workunit.client.0.smithi139.stderr:osd.0 is already down. 2024-09-15T21:02:47.264 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 allocmap 2024-09-15T21:02:47.298 INFO:tasks.workunit.client.0.smithi139.stdout:allocmap bluestore.allocmap 2024-09-15T21:02:50.365 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:02:50.359+0000 7fe60fe03ac0 -1 bluestore::NCB::operator()::(2)compare_allocators:: spillover 2024-09-15T21:02:50.366 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:02:50.359+0000 7fe60fe03ac0 -1 bluestore::NCB::compare_allocators::mismatch:: idx1=7 idx2=8 2024-09-15T21:02:50.366 INFO:tasks.workunit.client.0.smithi139.stderr:2024-09-15T21:02:50.359+0000 7fe60fe03ac0 -1 bluestore::NCB::read_allocation_from_drive_for_bluestore_tool::FAILURE. Allocator from file and allocator from metadata differ::ret=-1 2024-09-15T21:02:50.911 INFO:tasks.workunit.client.0.smithi139.stderr:allocmap failed: (1) Operation not permitted 2024-09-15T21:02:50.919 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: return 1 2024-09-15T21:02:50.920 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: return 1 2024-09-15T21:02:50.920 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: code=1 2024-09-15T21:02:50.920 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-bluefs-volume-ops 1 2024-09-15T21:02:50.920 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-15T21:02:50.920 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=1 2024-09-15T21:02:50.920 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-15T21:02:50.921 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-15T21:02:50.921 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-15T21:02:50.921 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-15T21:02:50.921 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-15T21:02:50.921 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-15T21:02:51.030 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-15T21:02:51.030 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-15T21:02:51.031 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T21:02:51.031 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-15T21:02:51.032 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T21:02:51.032 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-15T21:02:51.032 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-15T21:02:51.033 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:02:51.033 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-15T21:02:51.034 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-15T21:02:51.034 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T21:02:51.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:02:51.035 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 1 = 1 ']' 2024-09-15T21:02:51.036 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:181: teardown: '[' -n '' ']' 2024-09-15T21:02:51.036 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:185: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-09-15T21:02:51.036 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:186: teardown: mv td/osd-bluefs-volume-ops/bluestore_tool.log td/osd-bluefs-volume-ops/mgr.x.log td/osd-bluefs-volume-ops/mon.a.log td/osd-bluefs-volume-ops/osd.0.log /home/ubuntu/cephtest/archive/log 2024-09-15T21:02:51.037 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-15T21:02:51.041 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-15T21:02:51.041 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-15T21:02:51.041 INFO:tasks.workunit.client.0.smithi139.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.97169 2024-09-15T21:02:51.041 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.97169 2024-09-15T21:02:51.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:02:51.043 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-15T21:02:51.043 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-15T21:02:51.043 INFO:tasks.workunit.client.0.smithi139.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 1 2024-09-15T21:02:51.043 INFO:tasks.workunit:Stopping ['osd'] on client.0... 2024-09-15T21:02:51.043 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-15T21:02:51.342 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi139 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' 2024-09-15T21:02:51.534 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=0cbc9e452d814f998d67dafa34068a45 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi139 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' 2024-09-15T21:02:51.537 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T21:02:51.552 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T21:02:51.552 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T21:02:51.593 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T21:02:51.605 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T21:02:51.605 DEBUG:teuthology.orchestra.run.smithi139:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T21:02:51.625 INFO:teuthology.orchestra.run.smithi139.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T21:02:51.625 INFO:teuthology.orchestra.run.smithi139.stdout:============================================================================== 2024-09-15T21:02:51.625 INFO:teuthology.orchestra.run.smithi139.stdout:+hv01.front.sepi 67.205.162.81 3 u 61 64 377 0.076 -3.136 0.823 2024-09-15T21:02:51.625 INFO:teuthology.orchestra.run.smithi139.stdout:*hv02.front.sepi 63.231.80.2 3 u 52 64 377 0.086 -2.000 0.487 2024-09-15T21:02:51.625 INFO:teuthology.orchestra.run.smithi139.stdout:+hv03.front.sepi 74.6.168.72 3 u 51 64 377 0.089 -0.020 0.433 2024-09-15T21:02:51.625 INFO:teuthology.orchestra.run.smithi139.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-15T21:02:51.626 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T21:02:51.637 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T21:02:51.638 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T21:02:51.651 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T21:02:51.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T21:02:51.678 INFO:teuthology.task.internal:Duration was 2999.918430 seconds 2024-09-15T21:02:51.678 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T21:02:51.692 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T21:02:51.693 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T21:02:51.735 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T21:02:51.735 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi139.front.sepia.ceph.com 2024-09-15T21:02:51.735 DEBUG:teuthology.orchestra.run.smithi139:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T21:02:51.791 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T21:02:51.792 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T21:02:51.871 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T21:02:51.871 DEBUG:teuthology.orchestra.run.smithi139:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T21:02:51.968 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T21:02:51.984 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T21:02:51.984 DEBUG:teuthology.orchestra.run.smithi139:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T21:02:51.994 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T21:02:52.009 DEBUG:teuthology.orchestra.run.smithi139:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:02:52.050 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = core 2024-09-15T21:02:52.067 DEBUG:teuthology.orchestra.run.smithi139:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:02:52.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:02:52.115 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T21:02:52.127 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T21:02:52.129 DEBUG:teuthology.misc:Transferring archived files from smithi139:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905555/remote/smithi139 2024-09-15T21:02:52.129 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T21:02:55.004 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T21:02:55.004 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T21:02:55.070 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T21:02:55.079 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T21:02:55.079 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T21:02:55.092 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T21:02:55.093 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T21:02:55.117 INFO:teuthology.orchestra.run.smithi139.stdout: 658641 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 15 21:02 /home/ubuntu/cephtest 2024-09-15T21:02:55.117 INFO:teuthology.orchestra.run.smithi139.stdout: 788944 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 15 20:22 /home/ubuntu/cephtest/mnt.0 2024-09-15T21:02:55.118 INFO:teuthology.orchestra.run.smithi139.stdout: 799254 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 15 20:49 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T21:02:55.118 INFO:teuthology.orchestra.run.smithi139.stdout: 935721 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 15 20:49 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T21:02:55.118 INFO:teuthology.orchestra.run.smithi139.stdout: 935722 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Sep 15 21:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-09-15T21:02:55.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:02:55.121 INFO:teuthology.orchestra.run.smithi139.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-15T21:02:55.121 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi139 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-15T21:02:55.128 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T21:02:55.139 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T21:02:55.151 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi139 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-15T21:02:55.155 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} duration: 2999.918429851532 failure_reason: 'Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi139 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=0cbc9e452d814f998d67dafa34068a45 status: fail success: false 2024-09-15T21:02:55.155 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T21:02:55.258 INFO:teuthology.run:FAIL